[jira] [Created] (HADOOP-16409) Allow authoritative mode on non-qualified paths

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-16409:
--

 Summary: Allow authoritative mode on non-qualified paths
 Key: HADOOP-16409
 URL: https://issues.apache.org/jira/browse/HADOOP-16409
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Sean Mackrory
Assignee: Sean Mackrory


fs.s3a.authoritative.path currently requires a qualified URI (e.g. 
s3a://bucket/path) which is how I see this being used most immediately, but it 
also make sense for someone to just be able to configure /path, if all of their 
buckets follow that pattern, or if they're providing configuration already in a 
bucket-specific context (e.g. job-level configs, etc.) Just need to qualify 
whatever is passed in to allowAuthoritative to make that work.

Also, in HADOOP-16396 Gabor pointed out a few whitepace nits that I neglected 
to fix before merging.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

new committer: Gabor Bota

[Sean Mackrory]

The Project Management Committee (PMC) for Apache Hadoop
has invited Gabor Bota to become a committer and we are pleased
to announce that he has accepted.

Gabor has been working on the S3A file-system, especially on
the robustness and completeness of S3Guard to help deal with
inconsistency in object storage. I'm excited to see his work
with the community continue!

Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.

[jira] [Created] (HADOOP-16396) Allow authoritative mode on a subdirectory

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-16396:
--

 Summary: Allow authoritative mode on a subdirectory
 Key: HADOOP-16396
 URL: https://issues.apache.org/jira/browse/HADOOP-16396
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/s3
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Let's allow authoritative mode to be applied only to a subset of a bucket. This 
is coming primarily from a Hive warehousing use-case where Hive-managed tables 
can benefit from query planning, but can't speak for the rest of the bucket. 
This should be limited in scope and is not a general attempt to allow 
configuration on a per-path basis, as configuration is currently done on a 
per-process of a per-bucket basis.

I propose a new property (we could overload fs.s3a.metadatastore.authoritative, 
but that seems likely to cause confusion somewhere). A string would be allowed 
that would then be qualified in the context of the FileSystem, and used to 
check if it is a prefix for a given path. If it is, we act as though 
authoritative mode is enabled. If not, we revert to the existing behavior of 
fs.s3a.metadatastore.authoritative (which in practice will probably be false, 
the default, if the new property is in use).

Let's be clear about a few things:
* Currently authoritative mode only short-cuts the process to avoid a 
round-trip to S3 if we know it is safe to do so. This means that even when 
authoritative mode is enabled for a bucket, if the metadata store does not have 
a complete (or "authoritative") current listing cached, authoritative mode 
still has no effect. This will still apply.
* This will only apply to getFileStatus and listStatus, and internal calls to 
their internal counterparts. No other API is currently using authoritative mode 
to change behavior.
* This will only apply to getFileStatus and listStatus calls INSIDE the 
configured prefix. If there is a recursvie listing on the parent of the 
configured prefix, no change in behavior will be observed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16321) ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-16321.

Resolution: Fixed

> ITestS3ASSL+TestOpenSSLSocketFactory failing with 
> java.lang.UnsatisfiedLinkErrors
> -
>
> Key: HADOOP-16321
> URL: https://issues.apache.org/jira/browse/HADOOP-16321
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3, test
>Affects Versions: 3.3.0
> Environment: macos
>Reporter: Steve Loughran
>Assignee: Sahil Takiar
>Priority: Major
>
> the new test of HADOOP-16050  {{ITestS3ASSL}} is failing with 
> {{java.lang.UnsatisfiedLinkError}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16101) Use lighter-weight alternatives to innerGetFileStatus where possible

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-16101:
--

 Summary: Use lighter-weight alternatives to innerGetFileStatus 
where possible
 Key: HADOOP-16101
 URL: https://issues.apache.org/jira/browse/HADOOP-16101
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


Discussion in HADOOP-15999 highlighted the heaviness of a full 
innerGetFileStatus call, where many usages of it may need a lighter weight 
fileExists, etc. Let's investigate usage of innerGetFileStatus and slim it down 
where possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15819) S3A integration test failures: FileSystem is closed! - without parallel test run

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15819.

Resolution: Fixed

> S3A integration test failures: FileSystem is closed! - without parallel test 
> run
> 
>
> Key: HADOOP-15819
> URL: https://issues.apache.org/jira/browse/HADOOP-15819
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Affects Versions: 3.1.1
>Reporter: Gabor Bota
>Assignee: Adam Antal
>Priority: Critical
> Attachments: HADOOP-15819.000.patch, HADOOP-15819.001.patch, 
> HADOOP-15819.002.patch, S3ACloseEnforcedFileSystem.java, 
> S3ACloseEnforcedFileSystem.java, closed_fs_closers_example_5klines.log.zip
>
>
> Running the integration tests for hadoop-aws {{mvn -Dscale verify}} against 
> Amazon AWS S3 (eu-west-1, us-west-1, with no s3guard) we see a lot of these 
> failures:
> {noformat}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.408 
> s <<< FAILURE! - in 
> org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob
> [ERROR] 
> testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITDirectoryCommitMRJob)
>   Time elapsed: 0.027 s  <<< ERROR!
> java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 4.345 
> s <<< FAILURE! - in 
> org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob
> [ERROR] 
> testStagingDirectory(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob)
>   Time elapsed: 0.021 s  <<< ERROR!
> java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
> [ERROR] 
> testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJob)
>   Time elapsed: 0.022 s  <<< ERROR!
> java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.489 
> s <<< FAILURE! - in 
> org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest
> [ERROR] 
> testMRJob(org.apache.hadoop.fs.s3a.commit.staging.integration.ITStagingCommitMRJobBadDest)
>   Time elapsed: 0.023 s  <<< ERROR!
> java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.695 
> s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob
> [ERROR] testMRJob(org.apache.hadoop.fs.s3a.commit.magic.ITMagicCommitMRJob)  
> Time elapsed: 0.039 s  <<< ERROR!
> java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.015 
> s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory
> [ERROR] 
> testEverything(org.apache.hadoop.fs.s3a.commit.ITestS3ACommitterFactory)  
> Time elapsed: 0.014 s  <<< ERROR!
> java.io.IOException: s3a://cloudera-dev-gabor-ireland: FileSystem is closed!
> {noformat}
> The big issue is that the tests are running in a serial manner - no test is 
> running on top of the other - so we should not see that the tests are failing 
> like this. The issue could be in how we handle 
> org.apache.hadoop.fs.FileSystem#CACHE - the tests should use the same 
> S3AFileSystem so if A test uses a FileSystem and closes it in teardown then B 
> test will get the same FileSystem object from the cache and try to use it, 
> but it is closed.
> We see this a lot in our downstream testing too. It's not possible to tell 
> that the failed regression test result is an implementation issue in the 
> runtime code or a test implementation problem. 
> I've checked when and what closes the S3AFileSystem with a sightly modified 
> version of S3AFileSystem which logs the closers of the fs if an error should 
> occur. I'll attach this modified java file for reference. See the next 
> example of the result when it's running:
> {noformat}
> 2018-10-04 00:52:25,596 [Thread-4201] ERROR s3a.S3ACloseEnforcedFileSystem 
> (S3ACloseEnforcedFileSystem.java:checkIfClosed(74)) - Use after close(): 
> java.lang.RuntimeException: Using closed FS!.
>   at 
> org.apache.hadoop.fs.s3a.S3ACloseEnforcedFileSystem.checkIfClosed(S3ACloseEnforcedFileSystem.java:73)
>   at 
> org.apache.hadoop.fs.s3a.S3ACloseEnforcedFileSystem.mkdirs(S3ACloseEn

[jira] [Resolved] (HADOOP-15841) ABFS: change createRemoteFileSystemDuringInitialization default to true

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15841.

Resolution: Won't Fix

> ABFS: change createRemoteFileSystemDuringInitialization default to true
> ---
>
> Key: HADOOP-15841
> URL: https://issues.apache.org/jira/browse/HADOOP-15841
> Project: Hadoop Common
>  Issue Type: Sub-task
>    Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Major
>
> I haven't seen a way to create a working container (at least for the dfs 
> endpoint) except for setting 
> fs.azure.createRemoteFileSystemDuringInitialization=true. I personally don't 
> see that much of a downside to having it default to true, and it's a mild 
> inconvenience to remember to set it to true for some action to create a 
> container. I vaguely recall [~tmarquardt] considering changing this default 
> too.
> I propose we do it?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15999) [s3a] Better support for out-of-band operations

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15999:
--

 Summary: [s3a] Better support for out-of-band operations
 Key: HADOOP-15999
 URL: https://issues.apache.org/jira/browse/HADOOP-15999
 Project: Hadoop Common
  Issue Type: New Feature
Reporter: Sean Mackrory


S3Guard was initially done on the premise that a new MetadataStore would be the 
source of truth, and that it wouldn't provide guarantees if updates were done 
without using S3Guard.

I've been seeing increased demand for better support for scenarios where 
operations are done on the data that can't reasonably be done with S3Guard 
involved. For example:
* A file is deleted using S3Guard, and replaced by some other tool. S3Guard 
can't tell the difference between the new file and delete / list inconsistency 
and continues to treat the file as deleted.
* An S3Guard-ed file is overwritten by a longer file by some other tool. When 
reading the file, only the length of the original file is read.

We could possibly have smarter behavior here by querying both S3 and the 
MetadataStore (even in cases where we may currently only query the 
MetadataStore in getFileStatus) and use whichever one has the higher modified 
time.

This kills the performance boost we currently get in some workloads with the 
short-circuited getFileStatus, but we could keep it with authoritative mode 
which should give a larger performance boost. At least we'd get more 
correctness without authoritative mode and a clear declaration of when we can 
make the assumptions required to short-circuit the process. If we can't 
consider S3Guard the source of truth, we need to defer to S3 more.

We'd need to be extra sure of any locality / time zone issues if we start 
relying on mod_time more directly, but currently we're tracking the 
modification time as returned by S3 anyway.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15860) Trailing period in file names gets ignored for some operations.

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15860:
--

 Summary: Trailing period in file names gets ignored for some 
operations.
 Key: HADOOP-15860
 URL: https://issues.apache.org/jira/browse/HADOOP-15860
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


If you create a directory with a trailing period (e.g. '/test.') the period is 
silently dropped, and will be listed as simply '/test'. '/test.test' appears to 
work just fine.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15841) ABFS:

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15841:
--

 Summary: ABFS: 
 Key: HADOOP-15841
 URL: https://issues.apache.org/jira/browse/HADOOP-15841
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I haven't seen a way to create a working container (at least for the dfs 
endpoint) except for setting 
fs.azure.createRemoteFileSystemDuringInitialization=true. I personally don't 
see that much of a downside to having it default to true, and it's a mild 
inconvenience to remember to set it to true for some action to create a 
container. I vaguely recall [~tmarquardt] considering changing this default too.

I propose we do it?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15823) ABDS: Stop requiring client ID and tenant ID for MSI

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15823:
--

 Summary: ABDS: Stop requiring client ID and tenant ID for MSI
 Key: HADOOP-15823
 URL: https://issues.apache.org/jira/browse/HADOOP-15823
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Sean Mackrory


ABFS requires the user to configure the tenant ID and client ID. From my 
understanding of MSI, that shouldn't be necessary and is an added requirement 
compared to MSI in ADLS. Can that be dropped?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15797) optional / builtin modules confused for cloud storage

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15797.

Resolution: Won't Fix

> optional / builtin modules confused for cloud storage
> -
>
> Key: HADOOP-15797
> URL: https://issues.apache.org/jira/browse/HADOOP-15797
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/adl, fs/azure, fs/s3
>Affects Versions: 3.2.0, 3.1.1
>    Reporter: Sean Mackrory
>Priority: Major
>
> Throwing this in your .hadooprc results in hadoop-aws being in the classpath 
> but not hadoop-azure*:
> {quote}
> hadoop_add_to_classpath_tools hadoop-aws
> hadoop_add_to_classpath_tools hadoop-azure
> hadoop_add_to_classpath_tools hadoop-azure-datalake
> {quote}
> It would seem that the core issue is that that requires the module to have 
> listed it's dependencies in MODULE_NAME.tools-builtin.txt, whereas the Azure 
> connectors only have them listed in MODULE_NAME.tools-optional.txt. S3 does 
> both, and there's a comment in it's POM about how it needs to do this because 
> of the "hadoop s3guard" CLI.
> Maybe there's some history that I'm missing here, but I think what's wrong 
> here is that hadoop_add_to_classpath should get what it needs from optional 
> modules. builtin modules shouldn't even need hadoop_add_to_classpath to be 
> added anyway.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15797) optional / builtin modules confused for cloud storage

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15797:
--

 Summary: optional / builtin modules confused for cloud storage
 Key: HADOOP-15797
 URL: https://issues.apache.org/jira/browse/HADOOP-15797
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/adl, fs/azure, fs/s3
Reporter: Sean Mackrory


Throwing this in your .hadooprc results in hadoop-aws being in the classpath 
but not hadoop-azure*:

{quote}
hadoop_add_to_classpath_tools hadoop-aws
hadoop_add_to_classpath_tools hadoop-azure
hadoop_add_to_classpath_tools hadoop-azure-datalake
{quote}

It would seem that the core issue is that that requires the module to have 
listed it's dependencies in MODULE_NAME.tools-builtin.txt, whereas the Azure 
connectors only have them listed in MODULE_NAME.tools-optional.txt. S3 does 
both, and there's a comment in it's POM about how it needs to do this because 
of the "hadoop s3guard" CLI.

Maybe there's some history that I'm missing here, but I think what's wrong here 
is that hadoop_add_to_classpath should get what it needs from optional modules. 
builtin modules shouldn't even need hadoop_add_to_classpath to be added anyway.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HADOOP-15407 to trunk

[Sean Mackrory]

Thank you for the votes, everyone - we have surpassed the required number.
If anyone would like time to review this further this weekend, let me know
and I'll hold off. Otherwise I don't see a reason to wait, and I'll perform
the merge before Monday.

On Wed, Sep 19, 2018 at 9:15 PM Aaron Fabbri  wrote:

> +1 (Binding). Haven't thoroughly reviewed all the new code but
> successfully ran all the integration and unit tests today. Based on the
> progress this looks ready to merge.  Good work folks.
>
> On Wed, Sep 19, 2018 at 5:19 PM John Zhuge  wrote:
>
>> +1 (Binding)  Nice effort all around.
>>
>> On Tue, Sep 18, 2018 at 2:45 AM Steve Loughran 
>> wrote:
>>
>> > +1 (Binding)
>> >
>> > Ive been testing this; the current branch is rebased to trunk and all
>> the
>> > new tests are happy.
>> >
>> >
>> > the connector is as good as any of the connectors get before they are
>> > ready for people to play with: there are always surprises in the wild,
>> > usually network and configuration —those we have to wait and see what
>> > happens,
>> >
>> >
>> >
>> >
>> >
>> >
>> > > On 18 Sep 2018, at 04:10, Sean Mackrory  wrote:
>> > >
>> > > All,
>> > >
>> > > I would like to propose that HADOOP-15407 be merged to trunk. As
>> > described
>> > > in that JIRA, this is a complete reimplementation of the current
>> > > hadoop-azure storage driver (WASB) with some significant advantages.
>> The
>> > > impact outside of that module is very limited, however, and it appears
>> > that
>> > > on-going improvements will continue to be so. The tests have been
>> stable
>> > > for some time, and I believe we've reached the point of being ready
>> for
>> > > broader feedback and to continue incremental improvements in trunk.
>> > > HADOOP-15407 was rebased on trunk today and I had a successful test
>> run.
>> > >
>> > > I'd like to call out the contributions of Thomas Marquardt, Da Zhou,
>> and
>> > Steve
>> > > Loughran who have all contributed significantly to getting this
>> branch to
>> > > its current state. Numerous other developers are named in the commit
>> log
>> > > and the JIRA.
>> > >
>> > > I'll start us off:
>> > >
>> > > +1 (binding)
>> >
>> >
>>
>> --
>> John
>>
>

[jira] [Resolved] (HADOOP-15770) Merge HADOOP-15407 to trunk

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15770.

Resolution: Not A Problem

> Merge HADOOP-15407 to trunk
> ---
>
> Key: HADOOP-15770
> URL: https://issues.apache.org/jira/browse/HADOOP-15770
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>    Reporter: Sean Mackrory
>    Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-15770.001.patch, HADOOP-15770.002.patch
>
>
> I started a [VOTE] thread last night. Will be posting a patch of the current 
> branch state to get pre-commit checks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15773) Fix issues raised by Yetus

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15773:
--

 Summary: Fix issues raised by Yetus
 Key: HADOOP-15773
 URL: https://issues.apache.org/jira/browse/HADOOP-15773
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/azure
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I aggregated the HADOOP-15407 branch into a single patch and posted it on 
HADOOP-15770 just to get an aggregate report of all current issues raised by 
Yetus. There was a javac deprecation warning, a number of checkstyle issues, 
some whitespace issues, and there are a couple of valid javadoc errors I see 
locally. Let's fix them before we merge.

I see a number of wildcard imports, all for the contents of large classes of 
configuration constants, and I think those should stay the way they are. There 
are a number of existing checkstyle issues in WASB, too that are irrelevant for 
the merge, and there are some field visibility issues in tests that are 
required that way for the tests to work as designed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15770) Merge HADOOP-15407 to trunk

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15770:
--

 Summary: Merge HADOOP-15407 to trunk
 Key: HADOOP-15770
 URL: https://issues.apache.org/jira/browse/HADOOP-15770
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I started a [VOTE] thread last night. Will be posting a patch of the current 
branch state to get pre-commit checks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[VOTE] Merge HADOOP-15407 to trunk

[Sean Mackrory]

All,

I would like to propose that HADOOP-15407 be merged to trunk. As described
in that JIRA, this is a complete reimplementation of the current
hadoop-azure storage driver (WASB) with some significant advantages. The
impact outside of that module is very limited, however, and it appears that
on-going improvements will continue to be so. The tests have been stable
for some time, and I believe we've reached the point of being ready for
broader feedback and to continue incremental improvements in trunk.
HADOOP-15407 was rebased on trunk today and I had a successful test run.

I'd like to call out the contributions of Thomas Marquardt, Da Zhou, and Steve
Loughran who have all contributed significantly to getting this branch to
its current state. Numerous other developers are named in the commit log
and the JIRA.

I'll start us off:

+1 (binding)

Re: [Vote] Merge discussion for Node attribute support feature YARN-3409

[Sean Mackrory]

`mvn install` fails on trunk right now due to javadoc error in yarn-api. Is
that related? If so, can we revert or fix ASAP please?

On Wed, Sep 12, 2018 at 6:05 AM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Hi All,
>  Me and Sunil have successfully merged YARN-3409 to the trunk and
> we have created new umbrella jira YARN-8766 to track the pending jiras and
> the remaining planned improvement features.
> Thanks all for the support !
>
> Regards,
> + Naga
>
>
> On Wed, Sep 12, 2018 at 7:46 AM Naganarasimha Garla <
> naganarasimha...@apache.org> wrote:
>
> > Hi All,
> >  Voting has been running since 6 days and adding my vote we have
> 4
> > binding and 2 non binding +1's with no -1's this voting passes and we
> will
> > be merging the branch shortly. Thanks for all who participated in the
> > discussion and voting thread !
> >
> > Thanks and Regards,
> > + Naga
> >
> > On Mon, Sep 10, 2018 at 2:50 PM Zian Chen  wrote:
> >
> >> +1 for merge.
> >>
> >> > On Sep 9, 2018, at 10:47 PM, Weiwei Yang  wrote:
> >> >
> >> > +1 for the merge
> >> >
> >> > On Mon, Sep 10, 2018 at 12:06 PM Rohith Sharma K S <
> >> > rohithsharm...@apache.org> wrote:
> >> >
> >> >> +1 for merge
> >> >>
> >> >> -Rohith Sharma K S
> >> >>
> >> >> On Wed, 5 Sep 2018 at 18:01, Naganarasimha Garla <
> >> >> naganarasimha...@apache.org> wrote:
> >> >>
> >> >>> Hi All,
> >> >>>Thanks for feedback folks, based on the positive response
> >> >> starting
> >> >>> a Vote thread for merging YARN-3409 to master.
> >> >>>
> >> >>> Regards,
> >> >>> + Naga & Sunil
> >> >>>
> >> >>> On Wed, 5 Sep 2018 2:51 am Wangda Tan,  wrote:
> >> >>>
> >>  +1 for the merge, it gonna be a great addition to 3.2.0 release.
> >> Thanks
> >> >>> to
> >>  everybody for pushing this feature to complete.
> >> 
> >>  Best,
> >>  Wangda
> >> 
> >>  On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <
> >> >> bibin.chund...@huawei.com>
> >>  wrote:
> >> 
> >> > +1 for merge. Fetaure would be a good addition to 3.2 release.
> >> >
> >> > --
> >> > Bibin A Chundatt
> >> > M: +91-9742095715
> >> > E: bibin.chund...@huawei.com
> >> > 2012实验室-印研IT BU分部
> >> > 2012 Laboratories-IT BU Branch Dept.
> >> > From:Naganarasimha Garla
> >> > To:common-dev@hadoop.apache.org,Hdfs-dev,
> yarn-...@hadoop.apache.org
> >> ,
> >> > mapreduce-...@hadoop.apache.org,
> >> > Date:2018-08-29 20:00:44
> >> > Subject:[Discuss] Merge discussion for Node attribute support
> >> feature
> >> > YARN-3409
> >> >
> >> > Hi All,
> >> >
> >> > We would like to hear your thoughts on merging “Node Attributes
> >> >> Support
> >> >>> in
> >> > YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal
> is
> >> to
> >> >>> get
> >> > it in for HADOOP 3.2.
> >> >
> >> > *Major work happened in this branch*
> >> >
> >> > YARN-6858. Attribute Manager to store and provide node attributes
> in
> >> >> RM
> >> > YARN-7871. Support Node attributes reporting from NM to RM(
> >> >> distributed
> >> > node attributes)
> >> > YARN-7863. Modify placement constraints to support node attributes
> >> > YARN-7875. Node Attribute store for storing and recovering
> >> attributes
> >> >
> >> > *Detailed Design:*
> >> >
> >> > Please refer [1] for detailed design document.
> >> >
> >> > *Testing Efforts:*
> >> >
> >> > We did detailed tests for the feature in the last few weeks.
> >> > This feature will be enabled only when Node Attributes constraints
> >> are
> >> > specified through SchedulingRequest from AM.
> >> > Manager implementation will help to store and recover Node
> >> Attributes.
> >> > This
> >> > works with existing placement constraints.
> >> >
> >> > *Regarding to API stability:*
> >> >
> >> > All newly added @Public APIs are @Unstable.
> >> >
> >> > Documentation jira [3] could help to provide detailed
> configuration
> >> > details. This feature works from end-to-end and we tested this in
> >> our
> >> > local
> >> > cluster. Branch code is run against trunk and tracked via [4].
> >> >
> >> > We would love to get your thoughts before opening a voting thread.
> >> >
> >> > Special thanks to a team of folks who worked hard and contributed
> >> >>> towards
> >> > this efforts including design discussion / patch / reviews, etc.:
> >> >> Weiwei
> >> > Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali,
> >> >> Konstantinos
> >> > Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo,
> Chong
> >> >>> Chen.
> >> >
> >> > [1] :
> >> >
> >> >
> >> >>>
> >> >>
> >>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> >> > [2] : 

[jira] [Resolved] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15752.

Resolution: Duplicate

> [s3a] testDestroyNoBucket always fails
> --
>
> Key: HADOOP-15752
> URL: https://issues.apache.org/jira/browse/HADOOP-15752
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Reporter: Sean Mackrory
>Priority: Major
>
> ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way:
> {code}
> [INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> [ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 
> 73.198 s <<< FAILURE! - in 
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
> [ERROR] 
> testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  
> Time elapsed: 7.048 s  <<< FAILURE!
> java.lang.AssertionError: Expected a java.io.FileNotFoundException to be 
> thrown, but got the result: : 0
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15752) [s3a] testDestroyNoBucket always fails

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15752:
--

 Summary: [s3a] testDestroyNoBucket always fails
 Key: HADOOP-15752
 URL: https://issues.apache.org/jira/browse/HADOOP-15752
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Sean Mackrory


ITestS3GuardToolDynamoDB and ITestS3GuardToolLocal both fail the same way:

{code}
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[ERROR] Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 73.198 
s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal
[ERROR] 
testDestroyNoBucket(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal)  
Time elapsed: 7.048 s  <<< FAILURE!
java.lang.AssertionError: Expected a java.io.FileNotFoundException to be 
thrown, but got the result: : 0
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15745) Add ABFS configuration to ConfigRedactor

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15745:
--

 Summary: Add ABFS configuration to ConfigRedactor
 Key: HADOOP-15745
 URL: https://issues.apache.org/jira/browse/HADOOP-15745
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Sensitive information like credentials should be detected by ConfigRedactor so 
they never appear in logs or other channels. ABFS credentials are not all 
currently detected correctly so we should amend the default list of config 
patterns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15729:
--

 Summary: [s3a] stop treat fs.s3a.max.threads as the long-term 
minimum
 Key: HADOOP-15729
 URL: https://issues.apache.org/jira/browse/HADOOP-15729
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


A while ago the s3a connector started experiencing deadlocks because the AWS 
SDK requires an unbounded threadpool. It places monitoring tasks on the work 
queue before the tasks they wait on, so it's possible (has even happened with 
larger-than-default threadpools) for the executor to become permanently 
saturated and deadlock.

So we started giving an unbounded threadpool executor to the SDK, and using a 
bounded, blocking threadpool service for everything else S3A needs (although 
currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then 
only limits this threadpool, however we also specified fs.s3a.max.threads as 
the number of core threads in the unbounded threadpool, which in hindsight is 
pretty terrible.

Currently those core threads do not timeout, so this is actually setting a sort 
of minimum. Once that many tasks have been submitted, the threadpool will be 
locked at that number until it bursts beyond that, but it will only spin down 
that far. If fs.s3a.max.threads is set reasonably high and someone uses a bunch 
of S3 buckets, they could easily have thousands of idle threads constantly.

We should either not use fs.s3a.max.threads for the corepool size and introduce 
a new configuration, or we should simply allow core threads to timeout. I'm 
reading the OpenJDK source now to see what subtle differences there are between 
core threads and other threads if core threads can timeout.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15719) Fail-fast when using OAuth over http

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15719:
--

 Summary: Fail-fast when using OAuth over http
 Key: HADOOP-15719
 URL: https://issues.apache.org/jira/browse/HADOOP-15719
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


If you configure OAuth and then use abfs:// instead of abfss:// it will fail, 
but it takes a very long time, and still isn't very clear why. Good place to 
put a good human-readable exception message and fail fast.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15666) ABFS: Compatibility tests can fail

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15666.

Resolution: Not A Problem

Nope - I noticed at the end of last week it wasn't showing up any more.

> ABFS: Compatibility tests can fail
> --
>
> Key: HADOOP-15666
> URL: https://issues.apache.org/jira/browse/HADOOP-15666
> Project: Hadoop Common
>  Issue Type: Sub-task
>    Reporter: Sean Mackrory
>Priority: Major
>
> Sounds like other folks aren't hitting this, but I'm seeing failures in 2 
> compatibility test classes:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.417 
> s <<< FAILURE! - in 
> org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat
> [ERROR] 
> testBlobBackCompat(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat)
>   Time elapsed: 4.631 s  <<< ERROR!
> java.io.FileNotFoundException: 
> GET 
> http://ACCOUNT_NAME.dfs.core.windows.net/abfs-testcontainer-24b93630-e947-4139-8bbe-fcb96e61b7e4?resource=filesystem=5000=test/10=90=false
> StatusCode=404
> StatusDescription=The specified path does not exist.
> ErrorCode=PathNotFound
> ErrorMessage=The specified path does not exist.
> RequestId:cc2d3e74-d01f-007f-4911-30829500
> Time:2018-08-09T18:50:19.0627297Z
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:559)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:285)
> at 
> org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat.testBlobBackCompat(ITestAzureBlobFileSystemBackCompat.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: GET 
> http://ACCOUNT_NAME.dfs.core.windows.net/abfs-testcontainer-24b93630-e947-4139-8bbe-fcb96e61b7e4?resource=filesystem=5000=test/10=90=false
> StatusCode=404
> StatusDescription=The specified path does not exist.
> ErrorCode=PathNotFound
> ErrorMessage=The specified path does not exist.
> RequestId:cc2d3e74-d01f-007f-4911-30829500
> Time:2018-08-09T18:50:19.0627297Z
> at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:126)
> at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:151)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:404)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:282)
> ... 13 more
> {code}
> {code}
> [ERROR] Tests run: 5, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 
> 16.478 s <<< FAILURE! - in 
> org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
> [ERROR] 
> testListFileStatus(org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility)  
> Time elapsed: 6.334 s  <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at 
> org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility.testListFileStatus(ITestWasbAbfsCompatibility.java:80)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMe

[jira] [Created] (HADOOP-15694) ABFS: Allow OAuth credentials to not be tied to accounts

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15694:
--

 Summary: ABFS: Allow OAuth credentials to not be tied to accounts
 Key: HADOOP-15694
 URL: https://issues.apache.org/jira/browse/HADOOP-15694
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Now that there's OAuth support, it's possible to have a notion of identity 
that's distinct from the account itself. If a cluster is configured via OAuth 
with it's own identity, it's likely operators will want to use that identity 
regardless of which storage account a job uses.

So OAuth configs right now (and probably others) are looked up with 
.. I propose that we add a function for looking up these 
configs that returns an account-specific value if it exists, but in the event 
it does not will also try to return , if that exists.

I can work on a patch for this if nobody has any objections.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15688:
--

 Summary: ABFS: InputStream wrapped in FSDataInputStream
 Key: HADOOP-15688
 URL: https://issues.apache.org/jira/browse/HADOOP-15688
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I can't read Parquet files from ABFS. It has 2 different implementations to 
read seekable streams, and it'll use the one that uses ByteBuffer reads if it 
can. It currently decides to use the ByteBuffer read implementation because the 
FSDataInputStream it gets back wraps another FSDataInputStream, which 
implements ByteBufferReadable.

That's not the most robust way to check that ByteBufferReads are supported by 
the ultimately underlying InputStream, but it's unnecessary and probably a 
mistake to double-wrap the InputStream, so let's not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15636) Forgot to add ITestDynamoDBMetadataStore, now times out

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15636:
--

 Summary: Forgot to add ITestDynamoDBMetadataStore, now times out
 Key: HADOOP-15636
 URL: https://issues.apache.org/jira/browse/HADOOP-15636
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Gabor Bota


I committed HADOOP-14918 but I forgot to 'git add' the renamed test file. I 
would just add it and commit and reference the JIRA, but testTableProvision is 
now timing out, so we should look into that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15632) Address recent API incompatibilities in cloud storage

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15632:
--

 Summary: Address recent API incompatibilities in cloud storage
 Key: HADOOP-15632
 URL: https://issues.apache.org/jira/browse/HADOOP-15632
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory


I think these are unlikely to cause any real issues, but I had to sift through 
a lot of reports of binary incompatibility between Hadoop 2 and Hadoop 3 and 
I'd like that report to be smaller next time :)

HADOOP-14507 removes the single-parameter constructor from 2 classes that are 
marked as @Public and @Stable APIs. Propose we restore the original 
constructors and we document that they simply don't support per-bucket configs.

HADOOP-14935 changes getFileStatusInternal from protected to private. This 
differs from the proposed patch and isn't discussed in the review, so I assume 
it's just an accident.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15437) AzureBlobFS - Services

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15437.

Resolution: Duplicate

> AzureBlobFS - Services
> --
>
> Key: HADOOP-15437
> URL: https://issues.apache.org/jira/browse/HADOOP-15437
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15437-003.patch, HADOOP-15437.patch
>
>
> AzureBlobFS services and factories in the driver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15438) AzureBlobFS - Tests

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15438.

Resolution: Duplicate

> AzureBlobFS - Tests
> ---
>
> Key: HADOOP-15438
> URL: https://issues.apache.org/jira/browse/HADOOP-15438
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15438-001.patch, HADOOP-15438-003.patch
>
>
> AzureBlobFS functional and contract tests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15436) AzureBlobFS - Diagnostics and Utils

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15436.

Resolution: Duplicate

> AzureBlobFS - Diagnostics and Utils
> ---
>
> Key: HADOOP-15436
> URL: https://issues.apache.org/jira/browse/HADOOP-15436
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15436-001.patch, HADOOP-15436-003.patch
>
>
> AzureBlobFS Diagnostics and Utils classes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15435) AzureBlobFS - Constants

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15435.

Resolution: Duplicate

> AzureBlobFS - Constants
> ---
>
> Key: HADOOP-15435
> URL: https://issues.apache.org/jira/browse/HADOOP-15435
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15435-001.patch, HADOOP-15435-003.patch
>
>
> AzureBlobFS constants used across the driver.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15433) AzureBlobFS - Contracts

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15433.

Resolution: Duplicate

> AzureBlobFS - Contracts
> ---
>
> Key: HADOOP-15433
> URL: https://issues.apache.org/jira/browse/HADOOP-15433
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15433-001.patch, HADOOP-15433-003.patch
>
>
> All the internal, external contracts for the AzureBlobFS driver.
> Contracts include:
> - Configuration annotations
> - Configuration validation contract
> - Custom exceptions
> - Service contracts



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15432) AzureBlobFS - Base package classes and configuration files

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15432.

Resolution: Duplicate

> AzureBlobFS - Base package classes and configuration files
> --
>
> Key: HADOOP-15432
> URL: https://issues.apache.org/jira/browse/HADOOP-15432
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Esfandiar Manii
>Assignee: Esfandiar Manii
>Priority: Major
> Attachments: HADOOP-15432-001.patch, HADOOP-15432-003.patch
>
>
> Patch contains:
> - AzureBlobFileSystem and SecureAzureBlobFileSystem classes which are the 
> main interfaces Hadoop interacts with.
> - Updated Azure pom.xml with updated dependencies, updated parallel tests 
> configurations and maven shader plugin.
> - Checkstyle suppression file. Since http layer is generated automatically by 
> another libraries, it will not follow hadoop coding guidelines. Therefore a 
> few rules for checkstyles have been disabled.
> - Added test configuration file template to be used by the consumers. Similar 
> to wasb, all the configurations will go into this file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-14068.

   Resolution: Resolved
Fix Version/s: HADOOP-14918

We ended up carrying this across the finish line in HADOOP-14918.

> Add integration test version of TestMetadataStore for DynamoDB
> --
>
> Key: HADOOP-14068
> URL: https://issues.apache.org/jira/browse/HADOOP-14068
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1
>    Reporter: Sean Mackrory
>    Assignee: Sean Mackrory
>Priority: Major
> Fix For: HADOOP-14918
>
> Attachments: HADOOP-14068-HADOOP-13345.001.patch, 
> HADOOP-14068-HADOOP-13345.002.patch, HADOOP-14068-HADOOP-13345.003.patch, 
> HADOOP-14068-HADOOP-13345.004.patch, HADOOP-14068-HADOOP-13345.005.patch, 
> HADOOP-14068-HADOOP-13345.006.patch, HADOOP-14068-HADOOP-13345.007.patch, 
> HADOOP-14068-HADOOP-13345.008.patch
>
>
> I tweaked TestDynamoDBMetadataStore to run against the actual Amazon DynamoDB 
> service (as opposed to the "local" edition). Several tests failed because of 
> minor variations in behavior. I think the differences that are clearly 
> possible are enough to warrant extending that class as an ITest (but 
> obviously keeping the existing test so 99% of the the coverage remains even 
> when not configured for actual DynamoDB usage).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15541) AWS SDK can mistake stream timeouts for EOF and throw SdkClientExceptions

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15541:
--

 Summary: AWS SDK can mistake stream timeouts for EOF and throw 
SdkClientExceptions
 Key: HADOOP-15541
 URL: https://issues.apache.org/jira/browse/HADOOP-15541
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I've gotten a few reports of read timeouts not being handled properly in some 
Impala workloads. What happens is the following sequence of events (credit to 
Sailesh Mukil for figuring this out):
 * S3AInputStream.read() gets a SocketTimeoutException when it calls 
wrappedStream.read()
 * This is handled by onReadFailure -> reopen -> closeStream. When we try to 
drain the stream, SdkFilterInputStream.read() in the AWS SDK fails because of 
checkLength. The underlying Apache Commons stream returns -1 in the case of a 
timeout, and EOF.
 * The SDK assumes the -1 signifies an EOF, so assumes the bytes read must 
equal expected bytes, and because they don't (because it's a timeout and not an 
EOF) it throws an SdkClientException.

This is tricky to test for without a ton of mocking of AWS SDK internals, 
because you have to get into this conflicting state where the SDK has only read 
a subset of the expected bytes and gets a -1.

closeStream will abort the stream in the event of an IOException when draining. 
We could simply also abort in the event of an SdkClientException. I'm testing 
that this results in correct functionality in the workloads that seem to hit 
these timeouts a lot, but all the s3a tests continue to work with that change. 
I'm going to open an issue with the AWS SDK Github as well, but I'm not sure 
what the ideal outcome would be unless there's a good way to distinguish 
between a stream that has timed out and a stream that read all the data without 
huge rewrites.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15504) Upgrade Maven and Maven Wagon versions

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15504:
--

 Summary: Upgrade Maven and Maven Wagon versions
 Key: HADOOP-15504
 URL: https://issues.apache.org/jira/browse/HADOOP-15504
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I'm not even sure that Hadoop's combination of the relevant dependencies is 
vulnerable (even if they are, this is a relatively minor vulnerability), but 
this is at least showing up as an issue in automated vulnerability scans. 
Details can be found here [https://maven.apache.org/security.html] 
(CVE-2013-0253, CVE-2012-6153). Essentially the combination of maven 3.0.4 (we 
use 3.0, and I guess that maps to 3.0.4?) and older versions of wagon plugin 
don't use SSL properly.

I know some dependencies can be especially troublesome to upgrade - I suspect 
that Maven's critical role in our build might make this risky - so if anyone 
has ideas for how to more completely test this than a full build, please chime 
in,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15466) Correct units in adl.http.timeout

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15466:
--

 Summary: Correct units in adl.http.timeout
 Key: HADOOP-15466
 URL: https://issues.apache.org/jira/browse/HADOOP-15466
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/adl
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Comment in core-default.xml says seconds, but according to the SDK docs it's 
getting interpreted as milliseconds 
([https://github.com/Azure/azure-data-lake-store-java/blob/master/src/main/java/com/microsoft/azure/datalake/store/ADLStoreOptions.java#L139-L144).]
 Pinging [~ASikaria] for any additional comment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15442) ITestS3AMetrics.testMetricsRegister can't know metrics source's name

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15442:
--

 Summary: ITestS3AMetrics.testMetricsRegister can't know metrics 
source's name
 Key: HADOOP-15442
 URL: https://issues.apache.org/jira/browse/HADOOP-15442
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I've seen this test fail a bunch lately - mainly when the tests are all run 
(i.e. not individually) but not in parallel, it seems. If you dump out the 
sources when it fails, you see:

* The sources are numbered in the hundreds, so it's very unlikely that this 
actually gets the first one.
* The sources are numbered twice. There was logic to have the first one not be 
numbered, but that got messed up and now all sources are numbered twice, but 
the first one is only number once.

We could just remove the bad assertion, but then we're only testing the 
registry and not anything else about the way metrics flow all the way through 
the whole system. Worth it to fix the failing test, I think - knowing the 
source gets registered doesn't add a whole lot of value toward end-to-end 
metrics testing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15434) Upgrade to ADLS SDK that exposes current timeout

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15434:
--

 Summary: Upgrade to ADLS SDK that exposes current timeout
 Key: HADOOP-15434
 URL: https://issues.apache.org/jira/browse/HADOOP-15434
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


HADOOP-15356 aims to expose the ADLS SDK base timeout as a configurable option 
in Hadoop, but this isn't very testable without being able to read it back 
after the write logic has been applied.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Hadoop-trunk-Commit failing due to libprotoc version

[Sean Mackrory]

Most of the commit jobs in the last few hours have failed. I would suspect
a change in the machines or images used to run the job. Who has access to
confirm such a change and perhaps correct it?

[ERROR] Failed to execute goal
org.apache.hadoop:hadoop-maven-plugins:3.2.0-SNAPSHOT:protoc
(compile-protoc) on project hadoop-common:
org.apache.maven.plugin.MojoExecutionException: protoc version is
'libprotoc 2.6.1', expected version is '2.5.0' -> [Help 1]

[jira] [Created] (HADOOP-15316) GenericTestUtils can exceed maxSleepTime

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15316:
--

 Summary: GenericTestUtils can exceed maxSleepTime
 Key: HADOOP-15316
 URL: https://issues.apache.org/jira/browse/HADOOP-15316
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Probably shouldn't ever cause an issue, especially since Thread.sleep() can 
cause longer delays beyond your control anyway, but for larger values this 
could still behave unpredicatably in practice.
{code:java}
Thread.sleep(r.nextInt(maxSleepTime) + minSleepTime);
{code}
should be
{code:java}
Thread.sleep(r.nextInt(maxSleepTime - minSleepTime) + minSleepTime){code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15299) Bump Hadoop's Jackson 2 dependency 2.9.x

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15299:
--

 Summary: Bump Hadoop's Jackson 2 dependency 2.9.x
 Key: HADOOP-15299
 URL: https://issues.apache.org/jira/browse/HADOOP-15299
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.1.0, 3.2.0
Reporter: Sean Mackrory
Assignee: Sean Mackrory


There are a few new CVEs open against Jackson 2.7.x. It doesn't (necessarily) 
mean Hadoop is vulnerable to the attack - I don't know that it is, but fixes 
were released for 2.8.x and 2.9.x but not 2.7.x (which we're on). We shouldn't 
be on an unmaintained line, regardless. HBase is already on 2.9.x, we have a 
shaded client now, the API changes are relatively minor and so far in my 
testing I haven't seen any problems. I think many of our usual reasons to 
hesitate upgrading this dependency don't apply.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15199) hadoop_verify_confdir prevents previously valid log4j config file names

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15199:
--

 Summary: hadoop_verify_confdir prevents previously valid log4j 
config file names
 Key: HADOOP-15199
 URL: https://issues.apache.org/jira/browse/HADOOP-15199
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory


When starting a daemon, the shell scripts check that there's a log4j.properties 
file and logs an error if there isn't one. But there appear to be several 
instances of files named with a prefix or suffix (for example - I found this 
starting up HttpFS with httpfs-log4j.properties in a Bigtop-style deployment). 
We should probably loosen the check a little, something like this
{code:java}
diff --git 
a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh 
b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
index 2dc1dc8..df82bd2 100755
--- a/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
+++ b/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
@@ -651,7 +651,7 @@ function hadoop_verify_confdir
 {
   # Check only log4j.properties by default.
   # --loglevel does not work without logger settings in log4j.log4j.properties.
-  if [[ ! -f "${HADOOP_CONF_DIR}/log4j.properties" ]]; then
+  if [[ ! -f "${HADOOP_CONF_DIR}/*log4j*.properties" ]]; then
 hadoop_error "WARNING: log4j.properties is not found. HADOOP_CONF_DIR may 
be incomplete."
   fi
 }{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15161) s3a: Stream and common statistics missing from metrics

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15161:
--

 Summary: s3a: Stream and common statistics missing from metrics
 Key: HADOOP-15161
 URL: https://issues.apache.org/jira/browse/HADOOP-15161
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Input stream statistics aren't being passed through to metrics once merged. 
Also, the following "common statistics" are not being incremented or tracked by 
metrics:
{code}
OP_APPEND
OP_CREATE
OP_CREATE_NON_RECURSIVE
OP_DELETE
OP_GET_CONTENT_SUMMARY
OP_GET_FILE_CHECKSUM
OP_GET_STATUS
OP_MODIFY_ACL_ENTRIES
OP_OPEN
OP_REMOVE_ACL
OP_REMOVE_ACL_ENTRIES
OP_REMOVE_DEFAULT_ACL
OP_SET_ACL
OP_SET_OWNER
OP_SET_PERMISSION
OP_SET_TIMES
OP_TRUNCATE
{code}

Most of those make sense, but we can easily add OP_CREATE (and it's 
non-recursive cousin), OP_DELETE, OP_OPEN.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Sean Mackrory]

+1 (non-binding)

* Verified md5 of all artifacts
* Built with -Pdist
* Ran several s3a shell commands
* Started a pseudo-distributed HDFS and YARN cluster
* Ran grep and pi MR examples
* Sanity-checked contents of release notes, rat report, and changes

On Fri, Dec 8, 2017 at 1:31 PM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the efforts of all the contributors
> who contributed to this release, especially those who jumped on the issues
> found in RC0.
>
> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> fixed JIRAs since the previous 3.0.0-beta1 release.
>
> You can find the artifacts here:
>
> http://home.apache.org/~wang/3.0.0-RC1/
>
> I've done the traditional testing of building from the source tarball and
> running a Pi job on a single node cluster. I also verified that the shaded
> jars are not empty.
>
> Found one issue that create-release (probably due to the mvn deploy change)
> didn't sign the artifacts, but I fixed that by calling mvn one more time.
> Available here:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
>
> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> Pacific. My +1 to start.
>
> Best,
> Andrew
>

[jira] [Created] (HADOOP-15079) ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15079:
--

 Summary: ITestS3AFileOperationCost#testFakeDirectoryDeletion 
failing after OutputCommitter patch
 Key: HADOOP-15079
 URL: https://issues.apache.org/jira/browse/HADOOP-15079
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.1.0
Reporter: Sean Mackrory
Priority: Critical


I see this test failing with "object_delete_requests expected:<1> but was:<2>". 
I printed stack traces whenever this metric was incremented, and found the root 
cause to be that innerMkdirs is now causing two calls to delete fake 
directories when it previously caused only one. It is called once inside 
createFakeDirectory, and once directly inside innerMkdirs later:

{code}
at 
org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1625)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2634)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2599)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1498)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$11(S3AFileSystem.java:2684)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:108)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:259)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:230)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2682)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2657)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2021)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1956)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2305)
at 
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
at 
org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:209)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74
{code}

{code}
at 
org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1625)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2634)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2025)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1956)
at org.apache.hadoop.fs.FileSystem.mkdir

[jira] [Resolved] (HADOOP-14973) [s3a] Log StorageStatistics

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-14973.

Resolution: Duplicate

Resolving in favor of HADOOP-14475 since that's the right approach.

> [s3a] Log StorageStatistics
> ---
>
> Key: HADOOP-14973
> URL: https://issues.apache.org/jira/browse/HADOOP-14973
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1, 2.8.1
>    Reporter: Sean Mackrory
>    Assignee: Sean Mackrory
>
> S3A is currently storing much more detailed metrics via StorageStatistics 
> than are logged in a MapReduce job. Eventually, it would be nice to get 
> Spark, MapReduce and other workloads to retrieve and store these metrics, but 
> it may be some time before they all do that. I'd like to consider having S3A 
> publish the metrics itself in some form. This is tricky, as S3A has no daemon 
> but lives inside various other processes.
> Perhaps writing to a log file at some configurable interval and on close() 
> would be the best we could do. Other ideas would be welcome.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15055) Add s3 metrics from AWS SDK to s3a metrics tracking

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-15055:
--

 Summary: Add s3 metrics from AWS SDK to s3a metrics tracking
 Key: HADOOP-15055
 URL: https://issues.apache.org/jira/browse/HADOOP-15055
 Project: Hadoop Common
  Issue Type: New Feature
  Components: fs/s3
Reporter: Sean Mackrory
Assignee: Sean Mackrory






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] A final minor release off branch-2?

[Sean Mackrory]

>> You mentioned rolling-upgrades: It will be good to exactly outline the
type of testing. For e.g., the rolling-upgrades orchestration order has
direct implication on the testing done.

Complete details are available in HDFS-11096 where I'm trying to get
scripts to automate these tests committed so we can run them on Jenkins.
For HDFS, I follow the same order as the documentation. I did not see any
documentation indicate when to upgrade zkfc daemons, so it is done at the
end. I also did not see any documentation about a rolling upgrade for YARN,
so I'm doing ResourceManagers first then NodeManager, basically following
the pattern used in HDFS.

I can't speak much about app compatibility in YARN, etc. but the rolling
upgrade runs Terasuite from Hadoop 2 continually while doing the upgrade
and for sometime afterward. 1 incompatibility was found and fixed in trunk
quite a while ago - that part of the test has been working well for quite a
while now.

>> Copying data between 2.x clusters and 3.x clusters: Does this work
already? Is it broken anywhere that we cannot fix? Do we need bridging
features for this work?

HDFS-11096 also includes tests that data can be copied distcp'd over
webhdfs:// to and from old and new clusters regardless of where the distcp
job is launched from. I'll try a test run that uses hdfs:// this week, too.

As part of that JIRA I also looked through all the protobuf's for any
discrepancies / incompatibilities. One was found and fixed, but the rest
looked good to me.



On Mon, Nov 6, 2017 at 6:42 PM, Vinod Kumar Vavilapalli 
wrote:

> The main goal of the bridging release is to ease transition on stuff that
> is guaranteed to be broken.
>
> Of the top of my head, one of the biggest areas is application
> compatibility. When folks move from 2.x to 3.x, are their apps binary
> compatible? Source compatible? Or need changes?
>
> In 1.x -> 2.x upgrade, we did a bunch of work to atleast make old apps be
> source compatible. This means relooking at the API compatibility in 3.x and
> their impact of migrating applications. We will have to revist and
> un-deprecate old APIs, un-delete old APIs and write documentation on how
> apps can be migrated.
>
> Most of this work will be in 3.x line. The bridging release on the other
> hand will have deprecation for APIs that cannot be undeleted. This may be
> already have been done in many places. But we need to make sure and fill
> gaps if any.
>
> Other areas that I can recall from the old days
>  - Config migration: Many configs are deprecated or deleted. We need
> documentation to help folks to move. We also need deprecations in the
> bridging release for configs that cannot be undeleted.
>  - You mentioned rolling-upgrades: It will be good to exactly outline the
> type of testing. For e.g., the rolling-upgrades orchestration order has
> direct implication on the testing done.
>  - Story for downgrades?
>  - Copying data between 2.x clusters and 3.x clusters: Does this work
> already? Is it broken anywhere that we cannot fix? Do we need bridging
> features for this work?
>
> +Vinod
>
> > On Nov 6, 2017, at 12:49 PM, Andrew Wang 
> wrote:
> >
> > What are the known gaps that need bridging between 2.x and 3.x?
> >
> > From an HDFS perspective, we've tested wire compat, rolling upgrade, and
> > rollback.
> >
> > From a YARN perspective, we've tested wire compat and rolling upgrade.
> Arun
> > just mentioned an NM rollback issue that I'm not familiar with.
> >
> > Anything else? External to this discussion, these should be documented as
> > known issues for 3.0.
> >
> > Best.
> > Andrew
> >
> > On Sun, Nov 5, 2017 at 1:46 PM, Arun Suresh  wrote:
> >
> >> Thanks for starting this discussion VInod.
> >>
> >> I agree (C) is a bad idea.
> >> I would prefer (A) given that ATM, branch-2 is still very close to
> >> branch-2.9 - and it is a good time to make a collective decision to lock
> >> down commits to branch-2.
> >>
> >> I think we should also clearly define what the 'bridging' release should
> >> be.
> >> I assume it means the following:
> >> * Any 2.x user wanting to move to 3.x must first upgrade to the bridging
> >> release first and then upgrade to the 3.x release.
> >> * With regard to state store upgrades (at least NM state stores) the
> >> bridging state stores should be aware of all new 3.x keys so the
> implicit
> >> assumption would be that a user can only rollback from the 3.x release
> to
> >> the bridging release and not to the old 2.x release.
> >> * Use the opportunity to clean up deprecated API ?
> >> * Do we even want to consider a separate bridging release for 2.7, 2.8
> an
> >> 2.9 lines ?
> >>
> >> Cheers
> >> -Arun
> >>
> >> On Fri, Nov 3, 2017 at 5:07 PM, Vinod Kumar Vavilapalli <
> >> vino...@apache.org>
> >> wrote:
> >>
> >>> Hi all,
> >>>
> >>> With 3.0.0 GA around the corner (tx for the push, Andrew!), 2.9.0 RC
> out
> >>> (tx Arun / Subru!) and 2.8.2 (tx 

[jira] [Created] (HADOOP-14973) Log StorageStatistics

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14973:
--

 Summary: Log StorageStatistics
 Key: HADOOP-14973
 URL: https://issues.apache.org/jira/browse/HADOOP-14973
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Sean Mackrory
Assignee: Sean Mackrory


S3A is currently storing much more detailed metrics via StorageStatistics than 
are logged in a MapReduce job. Eventually, it would be nice to get Spark, 
MapReduce and other workloads to retrieve and store these metrics, but it may 
be some time before they all do that. I'd like to consider having S3A publish 
the metrics itself in some form. This is tricky, as S3A has no daemon but lives 
inside various other processes.

Perhaps writing to a log file at some configurable interval and on close() 
would be the best we could do. Other ideas would be welcome.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14972) Histogram metrics types for latency, etc.

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14972:
--

 Summary: Histogram metrics types for latency, etc.
 Key: HADOOP-14972
 URL: https://issues.apache.org/jira/browse/HADOOP-14972
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Sean Mackrory
Assignee: Sean Mackrory


We'd like metrics to track latencies for various operations, such as latencies 
for various request types, etc. This may need to be done different from current 
metrics types that are just counters of type long, and it needs to be done 
intelligently as these measurements are very numerous, and are primarily 
interesting due to the outliers that are unpredictably far from normal. 

* An adaptive, sparse histogram type. I envision something configurable with a 
maximumum granularity and a maximum number of bins. Initially, datapoints are 
tallied in bins with the maximum granularity. As we reach the maximum number of 
bins, bins are merged in even / odd pairs. There's some complexity here, 
especially to make it perform well and allow safe concurrency, but I like the 
ability to configure reasonable limits and retain as much granularity as 
possible without knowing the exact shape of the data beforehand.

* LongMetrics named "read_latency_600ms", "read_latency_800ms" to represent 
bins. This was suggested to me by [~fabbri]. I initially did not like the idea 
of having either so many hard-coded bins for however many op types, but this 
could also be done dynamically (we just hard-code which measurements we take, 
and with what granularity to group them, e.g. read_latency, 200 ms). The 
resulting dataset could be sparse and dynamic to allow for extreme outliers, 
but the granularity is still pre-determined.

* We could also simply track a certain number of the highest latencies, and 
basic descriptive statistics like a running average, min / max, etc. Inherently 
more limited in what it can show us, but much simpler and might still provide 
some insight when analyzing performance.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Merge HADOOP-13345 (S3Guard feature branch)

[Sean Mackrory]

+1 (non-binding)

On Tue, Aug 22, 2017 at 10:24 AM, Steve Loughran 
wrote:

> video being processed:  https://www.youtube.com/watch?
> v=oIe5Zl2YsLE=youtu.be
>
> its actually quite hard to show any benefits of s3guard on the command
> line, so I've ended up showing some scala tests where I turn on the
> (bundled) inconsistent AWS client to show how you then need to enable
> s3guard to make the stack traces go away
>
>
> On 22 Aug 2017, at 11:17, Steve Loughran > wrote:
>
> +1 (binding)
>
> I'm happy with it; it's a great piece of work by (in no particular order):
> Chris Nauroth, Aaron Fabbri, Sean McRory & Mingliang Liu. plus a few bits
> in the corners where I got to break things while they were all asleep. Also
> deserving a mention: Thomas Demoor & Ewan Higgs @ WDC for consultancy on
> the corners of S3, everyone who tested in (including our QA team), Sanjay
> Radia, & others.
>
> I've already done a couple of iterations of fixing checksyles & code
> reviews, so I think it is ready. I also have a branch-2 patch based on
> earlier work by Mingliang, for people who want that.
>
>
>
>
> On 17 Aug 2017, at 23:07, Aaron Fabbri  b...@cloudera.com>> wrote:
>
> Hello,
>
> I'd like to open a vote (7 days, ending August 24 at 3:10 PST) to merge the
> HADOOP-13345 feature branch into trunk.
>
> This branch contains the new S3Guard feature which adds metadata
> consistency features to the S3A client.  Formatted site documentation can
> be found here:
>
> https://github.com/apache/hadoop/blob/HADOOP-13345/
> hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md
>
> The current patch against trunk is posted here:
>
> https://issues.apache.org/jira/browse/HADOOP-13998
>
> The branch modifies the s3a portion of the hadoop-tools/hadoop-aws module:
>
> - The feature is off by default, and care has been taken to insure it has
> no impact when disabled.
> - S3Guard can be enabled with the production database which is backed by
> DynamoDB, or with a local, in-memory implementation that facilitates
> integration testing without having to pay for a database.
> - getFileStatus() as well as directory listing consistency has been
> implemented and thoroughly tested, including delete tracking.
> - Convenient Maven profiles for testing with and without S3Guard.
> - New failure injection code and integration tests that exercise it.  We
> use timers and a wrapper around the Amazon SDK client object to force
> consistency delays to occur.  This allows us to assert that S3Guard works
> as advertised.  This will be extended with more types of failure injection
> to continue hardening the S3A client.
>
> Outside of hadoop-tools/hadoop-aws's s3a directory there are some minor
> changes:
>
> - core-default.xml defaults and documentation for s3guard parameters.
> - A couple additional FS contract test cases around rename.
> - More goodies in LambdaTestUtils
> - A new CLI tool for inspecting and manipulating S3Guard features,
> including the backing MetadataStore database.
>
> This branch has seen extensive testing as well as use in production.  This
> branch makes significant improvements to S3A's test toolkit as well.
>
> Performance is typically on par with, and in some cases better than, the
> existing S3A code without S3Guard enabled.
>
> This feature was developed with contributions and feedback from many
> people.  I'd like to thank everyone who worked on HADOOP-13345 as well as
> all of those who contributed feedback and work on the original design
> document.
>
> This is the first major Apache Hadoop project I've worked on from start to
> finish, and I've really enjoyed it.  Please shout if I've missed anything
> important here or in the VOTE process.
>
> Cheers,
> Aaron Fabbri
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org common-dev-unsubscr...@hadoop.apache.org>
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org common-dev-h...@hadoop.apache.org>
>
>
>

[jira] [Created] (HADOOP-14717) Add StreamCapabilities support to s3a

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14717:
--

 Summary: Add StreamCapabilities support to s3a
 Key: HADOOP-14717
 URL: https://issues.apache.org/jira/browse/HADOOP-14717
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Reporter: Sean Mackrory
Assignee: Sean Mackrory






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Maven inconsistently overriding Zookeeper version

[Sean Mackrory]

There's some Maven magic going on here that I don't understand:
https://gist.github.com/mackrorysd/61c689f04c3595bcda9c256ec6b2da75

On line 2 of the gist, you can see me checking which ZooKeeper artifacts
get picked up when running dependency:tree with the ZooKeeper version
overridden with -Dzookeeper.version. It's all 3.5.3-beta, the version I'm
trying to override it to.

On line 84 of the gist, you can see me doing a clean build of Hadoop with
the same ZooKeeper version, but at the end it appears that
hadoop-yarn-server-resourcemanager is sometimes depending on 3.4.9 (the
version originally in the POM) and other times 3.5.3-beta.

I can't seem to work around that, or even explain it. Anybody have ideas?

[jira] [Created] (HADOOP-14585) Ensure controls in-place to prevent clients with significant clock skews pruning aggressively

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14585:
--

 Summary: Ensure controls in-place to prevent clients with 
significant clock skews pruning aggressively
 Key: HADOOP-14585
 URL: https://issues.apache.org/jira/browse/HADOOP-14585
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


>From discussion on HADOOP-14499:

{quote}
bear in mind that we can't guarantee that the clocks of all clients are in 
sync; you don't want a client whose TZ setting is wrong to aggressively prune 
things. Had that happen in production with files in shared filestore. This is 
why ant -diagnostics checks time consistency with temp files...
{quote}
{quote}
temp files work on a shared FS. AWS is actually somewhat sensitive to clocks: 
if your VM is too far out of time then auth actually fails, its ~+-15 minutes. 
There's some stuff in the Java SDK to actually calculate and adjust clock skew, 
presumably parsing the timestamp of a failure, calculating the difference and 
retrying. Which means that the field in SDKGlobalConfiguration could help 
identify the difference between local time and AWS time.
{quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14577) testGetFileStatus failing

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14577:
--

 Summary: testGetFileStatus failing
 Key: HADOOP-14577
 URL: https://issues.apache.org/jira/browse/HADOOP-14577
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


This test is failing for me when run individually or in parallel (with 
-Ds3guard). Even if I revert back to the commit that introduced it. I thought I 
had successful test runs on that before and haven't changed anything in my test 
configuration.

{code}Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.671 sec 
<<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AInconsistency
testGetFileStatus(org.apache.hadoop.fs.s3a.ITestS3AInconsistency)  Time 
elapsed: 4.475 sec  <<< FAILURE!
java.lang.AssertionError: S3Guard failed to list parent of inconsistent child.
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.hadoop.fs.s3a.ITestS3AInconsistency.testGetFileStatus(ITestS3AInconsistency.java:83){code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14576) DynamoDB tables may leave ACTIVE state after initial connection

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14576:
--

 Summary: DynamoDB tables may leave ACTIVE state after initial 
connection
 Key: HADOOP-14576
 URL: https://issues.apache.org/jira/browse/HADOOP-14576
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: HADOOP-13345
Reporter: Sean Mackrory


We currently only anticipate tables not being in the ACTIVE state when first 
connecting. It is possible for a table to be in the ACTIVE state and move to an 
UPDATING state during partitioning events. Attempts to read or write during 
that time will result in an AmazonServerException getting thrown. We should try 
to handle that better...



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14499) Findbufs warning in LocalMetadataStore

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14499:
--

 Summary: Findbufs warning in LocalMetadataStore
 Key: HADOOP-14499
 URL: https://issues.apache.org/jira/browse/HADOOP-14499
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


First saw this raised by Yetus on HADOOP-14433:
{code}
Bug type UC_USELESS_OBJECT (click for details)
In class org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore
In method org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.prune(long)
Value ancestors
Type java.util.LinkedList
At LocalMetadataStore.java:[line 300]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14489) ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be configured

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-14489.

Resolution: Fixed

Resolving as a duplicate. Thanks [~liuml07]

> ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be 
> configured
> 
>
> Key: HADOOP-14489
> URL: https://issues.apache.org/jira/browse/HADOOP-14489
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Sean Mackrory
>    Assignee: Sean Mackrory
>
> testConcurrentTableCreations fails with this: 
> {quote}java.lang.IllegalArgumentException: No DynamoDB table name 
> configured!{quote}
> I don't think that's necessary - should be able to shuffle stuff around and 
> either use the bucket name by default (like other DynamoDB tests would) or 
> use the table name that's configured later in the test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14489) ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be configured

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14489:
--

 Summary: ITestS3GuardConcurrentOps requires explicit DynamoDB 
table name to be configured
 Key: HADOOP-14489
 URL: https://issues.apache.org/jira/browse/HADOOP-14489
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


testConcurrentTableCreations fails with this: 
{quote}java.lang.IllegalArgumentException: No DynamoDB table name 
configured!{quote}

I don't think that's necessary - should be able to shuffle stuff around and 
either use the bucket name by default (like other DynamoDB tests would) or use 
the table name that's configured later in the test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14484) Ensure deleted parent directory tombstones are overwritten when implicitly recreated

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-14484.

Resolution: Duplicate

Resolving this as a duplicate, since I did end up doing it as part of the first 
patch, and it makes sense to continue to do so.

> Ensure deleted parent directory tombstones are overwritten when implicitly 
> recreated
> 
>
> Key: HADOOP-14484
> URL: https://issues.apache.org/jira/browse/HADOOP-14484
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>    Reporter: Sean Mackrory
>    Assignee: Sean Mackrory
>
> As discussed on HADOOP-13998, there may be a test missing (and possibly 
> broken metadata store implementations) for the case where a directory is 
> deleted but is later implicitly recreated by creating a file inside it, where 
> the tombstone is not overwritten. In such a case, listing the parent 
> directory would result in an error.
> This may also be happening because of HADOOP-14457, but we should add a test 
> for this other possibility anyway and fix it if it fails with any 
> implementations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14484) Ensure deleted parent directory tombstones are overwritten when implicitly recreated

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14484:
--

 Summary: Ensure deleted parent directory tombstones are 
overwritten when implicitly recreated
 Key: HADOOP-14484
 URL: https://issues.apache.org/jira/browse/HADOOP-14484
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


As discussed on HADOOP-13998, there may be a test missing (and possibly broken 
metadata store implementations) for the case where a directory is deleted but 
is later implicitly recreated by creating a file inside it, where the tombstone 
is not overwritten. In such a case, listing the parent directory would result 
in an error.

This may also be happening because of HADOOP-14457, but we should add a test 
for this other possibility anyway and fix it if it fails with any 
implementations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14457) LocalMetadataStore falsely reports empty parent directory authoritatively

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14457:
--

 Summary: LocalMetadataStore falsely reports empty parent directory 
authoritatively
 Key: HADOOP-14457
 URL: https://issues.apache.org/jira/browse/HADOOP-14457
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


Not a great test yet, but it at least reliably demonstrates the issue. 
LocalMetadataStore will sometimes erroneously report that a directory is empty 
with isAuthoritative = true when it *definitely* has children the metadatastore 
should know about. It doesn't appear to happen if the children are just 
directory. The fact that it's returning an empty listing is concerning, but the 
fact that it says it's authoritative *might* be a second bug.

{code}
diff --git 
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
 
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
index 78b3970..1821d19 100644
--- 
a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
+++ 
b/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
@@ -965,7 +965,7 @@ public boolean hasMetadataStore() {
   }
 
   @VisibleForTesting
-  MetadataStore getMetadataStore() {
+  public MetadataStore getMetadataStore() {
 return metadataStore;
   }
 
diff --git 
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
 
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
index 4339649..881bdc9 100644
--- 
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
+++ 
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRename.java
@@ -23,6 +23,11 @@
 import org.apache.hadoop.fs.contract.AbstractFSContract;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.Tristate;
+import org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata;
+import org.apache.hadoop.fs.s3a.s3guard.MetadataStore;
+import org.junit.Test;
 
 import static org.apache.hadoop.fs.contract.ContractTestUtils.dataset;
 import static org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset;
@@ -72,4 +77,24 @@ public void testRenameDirIntoExistingDir() throws Throwable {
 boolean rename = fs.rename(srcDir, destDir);
 assertFalse("s3a doesn't support rename to non-empty directory", rename);
   }
+
+  @Test
+  public void testMkdirPopulatesFileAncestors() throws Exception {
+final FileSystem fs = getFileSystem();
+final MetadataStore ms = ((S3AFileSystem) fs).getMetadataStore();
+final Path parent = path("testMkdirPopulatesFileAncestors/source");
+try {
+  fs.mkdirs(parent);
+  final Path nestedFile = new Path(parent, "dir1/dir2/dir3/file4");
+  byte[] srcDataset = dataset(256, 'a', 'z');
+  writeDataset(fs, nestedFile, srcDataset, srcDataset.length,
+  1024, false);
+
+  DirListingMetadata list = ms.listChildren(parent);
+  assertTrue("MetadataStore falsely reports authoritative empty list",
+  list.isEmpty() == Tristate.FALSE || !list.isAuthoritative());
+} finally {
+  fs.delete(parent, true);
+}
+  }
 }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14448) Play nice with ITestS3AEncryptionSSEC

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14448:
--

 Summary: Play nice with ITestS3AEncryptionSSEC
 Key: HADOOP-14448
 URL: https://issues.apache.org/jira/browse/HADOOP-14448
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: HADOOP-13345
Reporter: Sean Mackrory


HADOOP-14035 hasn't yet been merged with HADOOP-13345, but it adds tests that 
will break when run with S3Guard enabled. It expects that certain filesystem 
actions will throw exceptions when the client-provided encryption key is not 
configured properly, but those actions may sometimes bypass S3 entirely thanks 
to S3Guard (for example, getFileStatus may not actually need to invoke 
s3GetFileStatus). If the exception is never thrown, the test fails.

At a minimum we should tweak the tests so they definitely invoke S3 directly, 
or just skip the offending tests when anything but the Null implementation is 
in use. This also opens the larger question of whether or not S3Guard should be 
serving up metadata that is otherwise only accessible when an encryption key is 
provided.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14288) TestDynamoDBMetadataStore is broken unless we can fail faster without a table version

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14288:
--

 Summary: TestDynamoDBMetadataStore is broken unless we can fail 
faster without a table version
 Key: HADOOP-14288
 URL: https://issues.apache.org/jira/browse/HADOOP-14288
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Priority: Critical


HADOOP-14215 causes TestDynamoDBMetadataStore to fail because not having a 
version meta-entry in the table no longer fails immediately, but must timeout 
all the retries.

Let's make those configurable and have the test just turn them way down low.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14215) DynamoDB client should waitForActive on existing tables

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14215:
--

 Summary: DynamoDB client should waitForActive on existing tables
 Key: HADOOP-14215
 URL: https://issues.apache.org/jira/browse/HADOOP-14215
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I saw a case where 2 separate applications tried to use the same 
non-pre-existing table with table.create = true at about the same time. One 
failed with a ResourceInUse exception. If a table does not exist, we attempt to 
create it and then wait for it to enter the active state. If another jumps in 
in the middle of that, the table may exist, thus bypassing our call to 
waitForActive(), and then try to use the table immediately.

While we're at it, let's also make sure that the race condition where a table 
might get created between checking if it exists and attempting to create it is 
handled gracefully.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14181) Add validation of DynamoDB region

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14181:
--

 Summary: Add validation of DynamoDB region
 Key: HADOOP-14181
 URL: https://issues.apache.org/jira/browse/HADOOP-14181
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Currently you can configure a nonsensical AWS region for DynamoDB and it 
doesn't stop you - it'll just start using another region silently. I propose 
that we just check it's a valid region for the current SDK and throw an 
exception instead of proceeding.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14172) S3Guard: import does not import empty directory

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14172:
--

 Summary: S3Guard: import does not import empty directory
 Key: HADOOP-14172
 URL: https://issues.apache.org/jira/browse/HADOOP-14172
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


It imports everything comes up from listFiles, which includes only files (and 
their parent directories as a side-effect). My first thought on doing this 
would be to override S3AFileSystem to add an optional parameter to use 
AcceptAllButSelfAndS3nDirs instead of AcceptFilesOnly. But we could also 
manually traverse the tree to get all FileStatus objects directory by directory 
like we do for diff. That's far slower but doesn't add surface area to 
S3AFileSystem. But there's also the impact to other S3 clients to worry about - 
I could go either way on that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14168) S3GuardTool tests should not run if S3Guard is not set up

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14168:
--

 Summary: S3GuardTool tests should not run if S3Guard is not set up
 Key: HADOOP-14168
 URL: https://issues.apache.org/jira/browse/HADOOP-14168
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I saw ITestS3GuardToolDynamoDB fail when running without any S3Guard 
configuration set up because it will run even with -Ds3guard.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14158) Possible for modified configuration to leak into metadatastore in S3GuardTool

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14158:
--

 Summary: Possible for modified configuration to leak into 
metadatastore in S3GuardTool
 Key: HADOOP-14158
 URL: https://issues.apache.org/jira/browse/HADOOP-14158
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


It doesn't appear to do it when run from the command-line, but when running the 
S3GuardTool.run (i.e. the parent function of most of the functions used in the 
unit tests) from a unit test, you end up with a NullMetadataStore, regardless 
of what else was configured.

We create an instance of S3AFileSystem with the metadata store implementation 
overridden to NullMetadataStore so that we have distinct interfaces to S3 and 
the metadata store. S3Guard can later be called using this filesystem, causing 
it to pick up the filesystem's configuration, which instructs it to use the 
NullMetadataStore implementation. This shouldn't be possible.

It is unknown if this happens in any real-world scenario - I've been unable to 
reproduce the problem from the command-line. But it definitely happens in a 
test, it shouldn't, and fixing this will at least allow HADOOP-14145 to have an 
automated test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14023) S3Guard: DynamoDBMetadataStore logs nonsense region

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-14023.

Resolution: Resolved

Yes this is fixed. The root of the problem was an imperfect way of translating 
from endpoint to region, and there is now no codepath where the region isn't 
determined by bucket or by configuration directly.

> S3Guard: DynamoDBMetadataStore logs nonsense region
> ---
>
> Key: HADOOP-14023
> URL: https://issues.apache.org/jira/browse/HADOOP-14023
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: HADOOP-13345
>Reporter: Sean Mackrory
>    Assignee: Sean Mackrory
>Priority: Minor
> Attachments: HADOOP-14023-HADOOP-13345.001.patch
>
>
> When you specify a DynamoDB endpoint instead of it using the same one as the 
> S3 bucket, it thinks the region is "dynamodb":
> {code}
> > bin/hadoop s3a init -m dynamodb://region-test
> 2017-01-25 11:41:28,006 INFO s3guard.S3GuardTool: create metadata store: 
> dynamodb://region-test scheme: dynamodb
> 2017-01-25 11:41:28,116 WARN util.NativeCodeLoader: Unable to load 
> native-hadoop library for your platform... using builtin-java classes where 
> applicable
> 2017-01-25 11:41:29,421 INFO s3guard.DynamoDBMetadataStore: Creating 
> non-existent DynamoDB table region-test in region dynamodb
> 2017-01-25 11:41:34,637 INFO s3guard.S3GuardTool: Metadata store 
> DynamoDBMetadataStore{region=dynamodb, tableName=region-test} is initialized.
> {code}
> My guess is it's because of DynamoDBMetadataStore.initialize will call 
> getEndpointPrefix, and all the published endpoints for DynamoDB actually 
> start with dynamodb.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14145) Ensure GenericOptionParser is used for S3Guard CLI

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14145:
--

 Summary: Ensure GenericOptionParser is used for S3Guard CLI
 Key: HADOOP-14145
 URL: https://issues.apache.org/jira/browse/HADOOP-14145
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


As discussed in HADOOP-14094.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14129) ITestS3ACredentialsInURL sometimes fails

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14129:
--

 Summary: ITestS3ACredentialsInURL sometimes fails
 Key: HADOOP-14129
 URL: https://issues.apache.org/jira/browse/HADOOP-14129
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


This test sometimes fails. I believe it's expected that DynamoDB doesn't have 
access to the credentials if they're embedded in the URL instead of the 
configuration (and IMO that's fine - since the functionality hasn't been in 
previous releases and since we want to discourage this practice especially now 
that there are better alternatives). Weirdly, I only sometimes get this failure 
on the HADOOP-13345 branch. But if the problem turns out to be what I think it 
is, a simple Assume should fix it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14114) S3A can no longer handle unencoded + in URIs

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14114:
--

 Summary: S3A can no longer handle unencoded + in URIs 
 Key: HADOOP-14114
 URL: https://issues.apache.org/jira/browse/HADOOP-14114
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


Amazon secret access keys can include alphanumeric characters, but also / and + 
(I wish there was an official source that was really specific on what they can 
contain, but I'll have to rely on a few blog posts and my own experience).

Keys containing slashes used to be impossible to embed in the URL (e.g. 
s3a://access_key:secret_key@bucket/) but it is now possible to do it via URL 
encoding. Pluses used to work, but that is now *only* possible via URL encoding.

In the case of pluses, they don't appear to cause any other problems for 
parsing. So IMO the best all-around solution here is for people to URL-encode 
these keys always, but so that keys that used to work just fine can continue to 
work fine, all we need to do is detect that, log a warning, and we can 
re-encode it for the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-14090.

Resolution: Won't Fix
  Assignee: Sean Mackrory

> Allow users to specify region for DynamoDB table instead of endpoint
> 
>
> Key: HADOOP-14090
> URL: https://issues.apache.org/jira/browse/HADOOP-14090
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Sean Mackrory
>    Assignee: Sean Mackrory
> Attachments: HADOOP-14090-HADOOP-13345.001.patch
>
>
> Assuming the AWS SDK allows this, I think this would be a better way to 
> configure it for any usage on AWS itself (with endpoint still being an option 
> for AWS-compatible non-AWS use cases). Unless users actually care about a 
> specific endpoint, this is easier. Perhaps less important, HADOOP-14023 shows 
> that inferring the region from the endpoint (which granted, isn't that 
> necessary) doesn't work very well at all.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14094) Rethink S3GuardTool options

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14094:
--

 Summary: Rethink S3GuardTool options
 Key: HADOOP-14094
 URL: https://issues.apache.org/jira/browse/HADOOP-14094
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


I think we need to rework the S3GuardTool options. A couple of problems I've 
observed in the patches I've done on top of that and seeing other developers 
trying it out:

* We should probably wrap the current commands in an S3Guard-specific command, 
since 'init', 'destroy', etc. don't touch the buckets at all.
* Convert to whole-word options, as the single-letter options are already 
getting overloaded. Some patches I've submitted have added functionality where 
the obvious flag is already in use (e.g. -r for region, and read throughput, -m 
for minutes, and metadatastore uri).  I may do this early as part of 
HADOOP-14090.
* We have some options that must be in the config in some cases, and can be in 
the command in other cases. But I've seen someone try to specify the table name 
in the config and leave out the -m option, with no luck. Also, since commands 
hard-code table auto-creation, you might have configured table auto-creation, 
try to import to a non-existent table, and it tells you table auto-creation is 
off.

We need a more consistent policy for how things should get configured that 
addresses these problems and future-proofs the command a bit more.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14090:
--

 Summary: Allow users to specify region for DynamoDB table instead 
of endpoint
 Key: HADOOP-14090
 URL: https://issues.apache.org/jira/browse/HADOOP-14090
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


Assuming the AWS SDK allows this, I think this would be a better way to 
configure it for any usage on AWS itself (with endpoint still being an option 
for AWS-compatible non-AWS use cases). Unless users actually care about a 
specific endpoint, this is easier. Perhaps less important, HADOOP-14023 shows 
that inferring the region from the endpoint (which granted, isn't that 
necessary) doesn't work very well at all.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14085) Drop unnecessary type assertion and cast

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14085:
--

 Summary: Drop unnecessary type assertion and cast
 Key: HADOOP-14085
 URL: https://issues.apache.org/jira/browse/HADOOP-14085
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


This is a spin-off from some discussion on HADOOP-13736. There's an assertion 
and type-cast that often causes a large number of test failures (this can be 
seen at HADOOP-14041). It appears to be unnecessary - things work really well 
by just removing that, and unless there's a good reason or test of something 
S3-specific, we should handle FileStatus in metadata anyway.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14068:
--

 Summary: Add integration test version of TestMetadataStore for 
DynamoDB
 Key: HADOOP-14068
 URL: https://issues.apache.org/jira/browse/HADOOP-14068
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I tweaked TestDynamoDBMetadataStore to run against the actual Amazon DynamoDB 
service (as opposed to the "local" edition). Several tests failed because of 
minor variations in behavior. I think the differences that are clearly possible 
are enough to warrant extending that class as an ITest (but obviously keeping 
the existing test so 99% of the the coverage remains even when not configured 
for actual DynamoDB usage).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14046) Metastore destruction test creates table without version marker

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14046:
--

 Summary: Metastore destruction test creates table without version 
marker
 Key: HADOOP-14046
 URL: https://issues.apache.org/jira/browse/HADOOP-14046
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


ITestS3GuardToolDynamoDB.testDestroyDynamoDBMetadataStore fails after 
HADOOP-13985.

The gist of that test is that it creates a table directly in DynamoDB then 
tries to destroy it through S3Guard CLI tools. Since there's another test that 
tries to create a table through S3Guard CLI tools then deletes it directly in 
DynamoDB, I propose just combing the two, so S3Guard CLI tools are used for 
init and destroy, and we only use other tools to check for all the correct 
side-effects.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14027) Implicitly creating DynamoDB table ignores endpoint config

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14027:
--

 Summary: Implicitly creating DynamoDB table ignores endpoint config
 Key: HADOOP-14027
 URL: https://issues.apache.org/jira/browse/HADOOP-14027
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory
Assignee: Sean Mackrory


When you're using the 'bin/hadoop s3a init' command, it correctly uses the 
endpoint provided on the command-line (if provided), it will then use the 
endpoint in the config (if provided), and failing that it will default to the 
same region as the bucket.

However if you just set fs.s3a.s3guard.ddb.table.create to true and create a 
directory for a new bucket / table, it will always use the same region as the 
bucket, even if another endpoint is configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14023) S3Guard:

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-14023:
--

 Summary: S3Guard: 
 Key: HADOOP-14023
 URL: https://issues.apache.org/jira/browse/HADOOP-14023
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory


When you specify a DynamoDB endpoint instead of it using the same one as the S3 
bucket, it thinks the region is "dynamodb":

{code}
> bin/hadoop s3a init -m dynamodb://region-test
2017-01-25 11:41:28,006 INFO s3guard.S3GuardTool: create metadata store: 
dynamodb://region-test scheme: dynamodb
2017-01-25 11:41:28,116 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes where 
applicable
2017-01-25 11:41:29,421 INFO s3guard.DynamoDBMetadataStore: Creating 
non-existent DynamoDB table region-test in region dynamodb
2017-01-25 11:41:34,637 INFO s3guard.S3GuardTool: Metadata store 
DynamoDBMetadataStore{region=dynamodb, tableName=region-test} is initialized.
{code}

My guess is it's because of DynamoDBMetadataStore.initialize will call 
getEndpointPrefix, and all the published endpoints for DynamoDB actually start 
with dynamodb.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13868) Configure multi-part copies and uploads separately

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-13868:
--

 Summary: Configure multi-part copies and uploads separately
 Key: HADOOP-13868
 URL: https://issues.apache.org/jira/browse/HADOOP-13868
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


I've been looking at a big performance regression when writing to S3 from Spark 
that appears to have been introduced with HADOOP-12891.

In the Amazon SDK, the default threshold for multi-part copies is 320x the 
threshold for multi-part uploads (and the block size is 20x bigger), so I don't 
think it's wise for us 

I did some quick tests and it seems to me the sweet spot when multi-part copies 
start being faster is around 512MB. It wasn't as significant, but using 
104857600 (Amazon's default) for the blocksize was also slightly better.

I propose we do the following, although they're independent.

(1) Split the configuration. Ideally, I'd like to have 
fs.s3a.multipart.copy.threshold and fs.s3a.multipart.upload.threshold (and 
corresponding properties for the block size). But then there's the question of 
what to do with the existing fs.s3a.multipart.* properties. Deprecation? Leave 
it as a short-hand for configuring both (that's overridden by the more specific 
properties?).

(2) Consider increasing the default values. In my tests, 256 MB seemed to be 
where multipart uploads came into their own, and 512 MB was where multipart 
copies started outperforming the alternative. Would be interested to hear what 
other people have seen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13826) [s3a] Deadlock possible using Amazon S3 SDK

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-13826:
--

 Summary: [s3a] Deadlock possible using Amazon S3 SDK
 Key: HADOOP-13826
 URL: https://issues.apache.org/jira/browse/HADOOP-13826
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory


In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The 
TransferManager javadocs 
(http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html)
 explain how this is possible:

{quote}It is not recommended to use a single threaded executor or a thread pool 
with a bounded work queue as control tasks may submit subtasks that can't 
complete until all sub tasks complete. Using an incorrectly configured thread 
pool may cause a deadlock (I.E. the work queue is filled with control tasks 
that can't finish until subtasks complete but subtasks can't execute because 
the queue is filled).{quote}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Upgrading Hadoop dependencies and catching potential incompatibilities for HBase

[Sean Mackrory]

HBase folks,

I'd like to work on upgrading some of the dependencies in Hadoop that are
starting to lag behind:

jackson2 (https://issues.apache.org/jira/browse/HADOOP-12705)
jaxb-api (https://issues.apache.org/jira/browse/HADOOP-13659)
commons-configuration (https://issues.apache.org/jira/browse/HADOOP-13660)
htrace (https://issues.apache.org/jira/browse/HADOOP-13661)

Some of these (chiefly jackson) are known to be high risk upgrades and have
caused problems for HBase in the past. I wanted to give you a heads up
about this work and coordinate with you to make sure I do all the testing
you think is necessary to discover any potential problems well in advance
of the breakages.

I was going to look at just running HBase's unit tests when built against
my Hadoop's trunk with my changes (Should I expect any problems here? Is
there a branch already focused on Hadoop 3 compatibility or anything?) and
Bigtop's integration tests too. Anything else you would add to this?

[jira] [Created] (HADOOP-13662) Upgrade jackson2 version

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-13662:
--

 Summary: Upgrade jackson2 version
 Key: HADOOP-13662
 URL: https://issues.apache.org/jira/browse/HADOOP-13662
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Sean Mackrory
Assignee: Sean Mackrory


We're currently pulling in version 2.2.3 - I think we should upgrade to the 
latest 2.8.3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13661) Upgrade HTrace version

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-13661:
--

 Summary: Upgrade HTrace version
 Key: HADOOP-13661
 URL: https://issues.apache.org/jira/browse/HADOOP-13661
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Sean Mackrory


We're currently pulling in version 4.0.1-incubating - I think we should upgrade 
to the latest 4.1.0-incubating.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13660) Upgrade commons-configuration version

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-13660:
--

 Summary: Upgrade commons-configuration version
 Key: HADOOP-13660
 URL: https://issues.apache.org/jira/browse/HADOOP-13660
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Sean Mackrory
Assignee: Sean Mackrory


We're currently pulling in version 1.6 - I think we should upgrade to the 
latest 1.10.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13659) Upgrade jaxb-api version

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-13659:
--

 Summary: Upgrade jaxb-api version
 Key: HADOOP-13659
 URL: https://issues.apache.org/jira/browse/HADOOP-13659
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Sean Mackrory
Assignee: Sean Mackrory


We're currently pulling in version 2.2.2 - I think we should upgrade to the 
latest 2.2.12.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13629) Make RemoteEditLogManifest.committedTxnId optional in Protocol Buffers

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-13629:
--

 Summary: Make RemoteEditLogManifest.committedTxnId optional in 
Protocol Buffers
 Key: HADOOP-13629
 URL: https://issues.apache.org/jira/browse/HADOOP-13629
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


HDFS-10519 introduced a new field in the RemoteEditLogManifest message. It can 
be made optional to improve wire-compatibility with previous versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-12667) s3a: Support createNonRecursive API

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-12667:
--

 Summary: s3a: Support createNonRecursive API
 Key: HADOOP-12667
 URL: https://issues.apache.org/jira/browse/HADOOP-12667
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Sean Mackrory
Assignee: Sean Mackrory


HBase and other clients rely on the createNonRecursive API, which was recently 
un-deprecated. S3A currently does not support it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-12346) Increase some default timeouts / retries for S3a connector

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-12346:
--

 Summary: Increase some default timeouts / retries for S3a connector
 Key: HADOOP-12346
 URL: https://issues.apache.org/jira/browse/HADOOP-12346
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Reporter: Sean Mackrory


I've been seeing some flakiness in jobs runnings against S3a, both first hand 
and with other accounts, for which increasing fs.s3a.connection.timeout and 
fs.s3a.attempts.maximum have been a reliable solution. I propose we increase 
the defaults.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HADOOP-9641) hadoop-client should not exclude commons-httpclient

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-9641:
-

 Summary: hadoop-client should not exclude commons-httpclient
 Key: HADOOP-9641
 URL: https://issues.apache.org/jira/browse/HADOOP-9641
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


The hadoop-client project excludes commons-httpclient from it's dependencies. I 
believe this is wrong, because it's necessary to run LocalJobRunner (e.g. 
submit jobs from Eclipse) and manually including it on the classpath from 
another location allows the client to succeed without any apparent problems.

I've been unable to determine the original reason for the exclusion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-9641) hadoop-client should not exclude commons-httpclient

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-9641.
---

Resolution: Duplicate

 hadoop-client should not exclude commons-httpclient
 ---

 Key: HADOOP-9641
 URL: https://issues.apache.org/jira/browse/HADOOP-9641
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory

 The hadoop-client project excludes commons-httpclient from it's dependencies. 
 I believe this is wrong, because it's necessary to run LocalJobRunner (e.g. 
 submit jobs from Eclipse) and manually including it on the classpath from 
 another location allows the client to succeed without any apparent problems.
 I've been unable to determine the original reason for the exclusion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9468) JVM path embedded in fuse binaries

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-9468:
-

 Summary: JVM path embedded in fuse binaries
 Key: HADOOP-9468
 URL: https://issues.apache.org/jira/browse/HADOOP-9468
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
Assignee: Sean Mackrory


When the FUSE binaries are built, the paths to libraries in the JVMs is 
embedded in the RPATH so that they can be found at run-time. From an Apache 
Bigtop perspective, this is not sufficient because the software may be run on a 
machine configured very differently from the one on which they were built - so 
a wrapper sets LD_LIBRARY_PATH according to JAVA_HOME. I recently saw an issue 
where the original JVM path existed, causing LD_LIBRARY_PATH to be ignored in 
favor of the RPATH, but it was not the JVM intended for running Hadoop (not 
JAVA_HOME), and this caused problems.

I'm told that setting LD_LIBRARY_PATH is standard practice before using the 
fuse program anyway, and if that's the case, I think removing the RPATH from 
the binaries is a good idea.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8795) BASH tab completion doesn't look in PATH, assumes path to executable is specified

[Sean Mackrory (JIRA)]

Sean Mackrory created HADOOP-8795:
-

 Summary: BASH tab completion doesn't look in PATH, assumes path to 
executable is specified
 Key: HADOOP-8795
 URL: https://issues.apache.org/jira/browse/HADOOP-8795
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sean Mackrory
 Attachments: HADOOP-8795.patch

bash-tab-completion/hadoop.sh checks that the first token in the command is an 
existing, executable file - which assumes that the path to the hadoop 
executable is specified (or that it's in the working directory). If the 
executable is somewhere else in PATH, tab completion will not work.

I propose that the first token be passed through 'which' so that any 
executables in the path also get detected. I've tested that this technique will 
work in the event that relative and absolute paths are used as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira