[jira] [Commented] (HADOOP-16424) S3Guard fsck: Check internal consistency of the MetadataStore

2019-10-17 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953649#comment-16953649 ] Gabor Bota commented on HADOOP-16424: - Our code should not be creating orphan entries. If we have

[jira] [Comment Edited] (HADOOP-16424) S3Guard fsck: Check internal consistency of the MetadataStore

2019-10-17 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898870#comment-16898870 ] Gabor Bota edited comment on HADOOP-16424 at 10/17/19 11:35 AM: Tasks

[jira] [Comment Edited] (HADOOP-16484) S3A to warn or fail if S3Guard is disabled

2019-10-16 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952938#comment-16952938 ] Gabor Bota edited comment on HADOOP-16484 at 10/16/19 3:29 PM: --- I removed

[jira] [Commented] (HADOOP-16484) S3A to warn or fail if S3Guard is disabled

2019-10-16 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952938#comment-16952938 ] Gabor Bota commented on HADOOP-16484: - I removed status (merged with inform) so at inform level it

[jira] [Commented] (HADOOP-16653) S3Guard DDB overreacts to no tag access

2019-10-14 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951043#comment-16951043 ] Gabor Bota commented on HADOOP-16653: - It was cleary in the docs, so I'll update that as well:

[jira] [Comment Edited] (HADOOP-16653) S3Guard DDB overreacts to no tag access

2019-10-14 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951043#comment-16951043 ] Gabor Bota edited comment on HADOOP-16653 at 10/14/19 2:47 PM: --- It was

[jira] [Commented] (HADOOP-16349) DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

2019-10-11 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949341#comment-16949341 ] Gabor Bota commented on HADOOP-16349: - https://github.com/apache/hadoop/pull/1576 Fixed in

[jira] [Resolved] (HADOOP-16349) DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

2019-10-11 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16349. - Resolution: Fixed > DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

[jira] [Resolved] (HADOOP-16520) Race condition in DDB table init and waiting threads

2019-10-11 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16520. - Resolution: Fixed > Race condition in DDB table init and waiting threads >

[jira] [Commented] (HADOOP-16520) Race condition in DDB table init and waiting threads

2019-10-11 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949339#comment-16949339 ] Gabor Bota commented on HADOOP-16520: - +1 on #1576 from [~ste...@apache.org]. Committing. Thanks.

[jira] [Updated] (HADOOP-16520) Race condition in DDB table init and waiting threads

2019-10-11 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16520: Summary: Race condition in DDB table init and waiting threads (was: race condition in DDB table

[jira] [Commented] (HADOOP-16349) DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

2019-10-01 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942161#comment-16942161 ] Gabor Bota commented on HADOOP-16349: - I'm going to fix this with HADOOP-16520 >

[jira] [Work started] (HADOOP-16349) DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

2019-10-01 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16349 started by Gabor Bota. --- > DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry >

[jira] [Assigned] (HADOOP-16349) DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry

2019-10-01 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16349: --- Assignee: Gabor Bota > DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn

[jira] [Work started] (HADOOP-16520) race condition in DDB table init and waiting threads

2019-09-28 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16520 started by Gabor Bota. --- > race condition in DDB table init and waiting threads >

[jira] [Commented] (HADOOP-16520) race condition in DDB table init and waiting threads

2019-09-26 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938690#comment-16938690 ] Gabor Bota commented on HADOOP-16520: - I'll try to solve this the other way: add the version marker

[jira] [Assigned] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop

2019-09-25 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16579: --- Assignee: Norbert Kalmar > Upgrade to Apache Curator 4.2.0 in Hadoop >

[jira] [Resolved] (HADOOP-16529) Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property

2019-09-25 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16529. - Resolution: Workaround > Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set

[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-09-25 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937551#comment-16937551 ] Gabor Bota commented on HADOOP-16138: - That is true. Sorry [~ayushtkn]. The way I plan to fix it:

[jira] [Resolved] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-09-23 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16138. - Target Version/s: 3.3.0 Resolution: Fixed > hadoop fs mkdir / of nonexistent abfs

[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-09-23 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16935762#comment-16935762 ] Gabor Bota commented on HADOOP-16138: - +1 by [~ste...@apache.org] on PR #1302. Committing. >

[jira] [Updated] (HADOOP-16565) Region must be provided when requesting session credentials or SdkClientException will be thrown

2019-09-23 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16565: Resolution: Fixed Status: Resolved (was: Patch Available) > Region must be provided

[jira] [Resolved] (HADOOP-16547) s3guard prune command doesn't get AWS auth chain from FS

2019-09-18 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16547. - Resolution: Fixed > s3guard prune command doesn't get AWS auth chain from FS >

[jira] [Commented] (HADOOP-16547) s3guard prune command doesn't get AWS auth chain from FS

2019-09-18 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932692#comment-16932692 ] Gabor Bota commented on HADOOP-16547: - +1 on PR 1402. Committed. > s3guard prune command doesn't

[jira] [Assigned] (HADOOP-16484) S3A to warn or fail if S3Guard is disabled

2019-09-18 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16484: --- Assignee: Gabor Bota > S3A to warn or fail if S3Guard is disabled >

[jira] [Work stopped] (HADOOP-16520) race condition in DDB table init and waiting threads

2019-09-18 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16520 stopped by Gabor Bota. --- > race condition in DDB table init and waiting threads >

[jira] [Commented] (HADOOP-16565) Region must be provided when requesting session credentials or SdkClientException will be thrown

2019-09-18 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932216#comment-16932216 ] Gabor Bota commented on HADOOP-16565: - +1 from [~ste...@apache.org] on PR 1454. Committing. >

[jira] [Updated] (HADOOP-16565) Region must be provided when requesting session credentials or SdkClientException will be thrown

2019-09-16 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16565: Status: Patch Available (was: In Progress) > Region must be provided when requesting session

[jira] [Work started] (HADOOP-16565) Region must be provided when requesting session credentials or SdkClientException will be thrown

2019-09-16 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16565 started by Gabor Bota. --- > Region must be provided when requesting session credentials or >

[jira] [Updated] (HADOOP-16565) Region must be provided when requesting session credentials or SdkClientException will be thrown

2019-09-16 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16565: Summary: Region must be provided when requesting session credentials or SdkClientException will

[jira] [Reopened] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."

2019-09-13 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reopened HADOOP-16565: - Reopened to add a message. > Fix "com.amazonaws.SdkClientException: Unable to find a region via

[jira] [Work started] (HADOOP-16520) race condition in DDB table init and waiting threads

2019-09-13 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16520 started by Gabor Bota. --- > race condition in DDB table init and waiting threads >

[jira] [Assigned] (HADOOP-16520) race condition in DDB table init and waiting threads

2019-09-13 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16520: --- Assignee: Gabor Bota > race condition in DDB table init and waiting threads >

[jira] [Resolved] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."

2019-09-13 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16565. - Resolution: Workaround > Fix "com.amazonaws.SdkClientException: Unable to find a region via

[jira] [Commented] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."

2019-09-13 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929284#comment-16929284 ] Gabor Bota commented on HADOOP-16565: - It worked with the following STS config: {noformat}

[jira] [Commented] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."

2019-09-13 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929242#comment-16929242 ] Gabor Bota commented on HADOOP-16565: - I don't have any STS endpoint besides {noformat}

[jira] [Resolved] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16566. - Resolution: Fixed > S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of >

[jira] [Commented] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928734#comment-16928734 ] Gabor Bota commented on HADOOP-16566: - +1 by [~ste...@apache.org] on #1433 PR. Committing. >

[jira] [Resolved] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16423. - Resolution: Fixed > S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log) >

[jira] [Commented] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928693#comment-16928693 ] Gabor Bota commented on HADOOP-16566: - maybe, but it would be better to update guava everywhere. >

[jira] [Work started] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16566 started by Gabor Bota. --- > S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of >

[jira] [Created] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch

2019-09-12 Thread Gabor Bota (Jira)
Gabor Bota created HADOOP-16566: --- Summary: S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch Key: HADOOP-16566 URL: https://issues.apache.org/jira/browse/HADOOP-16566

[jira] [Updated] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16566: Component/s: fs/s3 > S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of >

[jira] [Updated] (HADOOP-16566) S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of com.google.common.base.Stopwatch

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16566: Affects Version/s: 3.3.0 > S3Guard fsck: Use org.apache.hadoop.util.StopWatch instead of >

[jira] [Created] (HADOOP-16565) Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain."

2019-09-12 Thread Gabor Bota (Jira)
Gabor Bota created HADOOP-16565: --- Summary: Fix "com.amazonaws.SdkClientException: Unable to find a region via the region provider chain." Key: HADOOP-16565 URL: https://issues.apache.org/jira/browse/HADOOP-16565

[jira] [Commented] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-09-12 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928449#comment-16928449 ] Gabor Bota commented on HADOOP-16423: - +1 from [~ste...@apache.org] on PR #1208. Committing.

[jira] [Created] (HADOOP-16564) S3Guarld fsck: Add docs to the first iteration (S3->ddbMS, -verify)

2019-09-12 Thread Gabor Bota (Jira)
Gabor Bota created HADOOP-16564: --- Summary: S3Guarld fsck: Add docs to the first iteration (S3->ddbMS, -verify) Key: HADOOP-16564 URL: https://issues.apache.org/jira/browse/HADOOP-16564 Project: Hadoop

[jira] [Created] (HADOOP-16563) S3Guard fsck: Detect if a directory if authoritative and highlight errors if detected in it

2019-09-12 Thread Gabor Bota (Jira)
Gabor Bota created HADOOP-16563: --- Summary: S3Guard fsck: Detect if a directory if authoritative and highlight errors if detected in it Key: HADOOP-16563 URL: https://issues.apache.org/jira/browse/HADOOP-16563

[jira] [Commented] (HADOOP-16507) S3Guard fsck: Add option to configure severity (level) for the scan

2019-09-10 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926405#comment-16926405 ] Gabor Bota commented on HADOOP-16507: - ++ additional task: Add `-verbose` arg to the fsck with

[jira] [Commented] (HADOOP-16550) Spark config name error on the Launching Applications Using Docker Containers page

2019-09-06 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924058#comment-16924058 ] Gabor Bota commented on HADOOP-16550: - Merged PR #9 > Spark config name error on the Launching

[jira] [Resolved] (HADOOP-16550) Spark config name error on the Launching Applications Using Docker Containers page

2019-09-06 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16550. - Resolution: Fixed > Spark config name error on the Launching Applications Using Docker

[jira] [Updated] (HADOOP-16550) Spark config name error on the Launching Applications Using Docker Containers page

2019-09-05 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16550: Description: On the "Launching Applications Using Docker Containers" page at the "Example:

[jira] [Commented] (HADOOP-16550) Spark config name error on the Launching Applications Using Docker Containers page

2019-09-05 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923501#comment-16923501 ] Gabor Bota commented on HADOOP-16550: - Sure, thanks for the contribution [~attilapiros]. LGTM, +1.

[jira] [Commented] (HADOOP-16529) Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property

2019-08-22 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913420#comment-16913420 ] Gabor Bota commented on HADOOP-16529: - Note: a possible fix for this could be to add {code:java}

[jira] [Created] (HADOOP-16529) Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property

2019-08-22 Thread Gabor Bota (Jira)
Gabor Bota created HADOOP-16529: --- Summary: Allow AZURE_CREATE_REMOTE_FILESYSTEM_DURING_INITIALIZATION to be set from abfs.xml property Key: HADOOP-16529 URL: https://issues.apache.org/jira/browse/HADOOP-16529

[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-08-22 Thread Gabor Bota (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913415#comment-16913415 ] Gabor Bota commented on HADOOP-16138: - I found out what's happening. We auto-create all containers

[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-08-15 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908308#comment-16908308 ] Gabor Bota commented on HADOOP-16138: - Based on offline discussion with [~mackrorysd] we agreed

[jira] [Updated] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-08-15 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16138: Status: In Progress (was: Patch Available) > hadoop fs mkdir / of nonexistent abfs container

[jira] [Updated] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-08-15 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16138: Status: Patch Available (was: In Progress) > hadoop fs mkdir / of nonexistent abfs container

[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-08-15 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908062#comment-16908062 ] Gabor Bota commented on HADOOP-16138: - So I've created a test for it (new test class for testing

[jira] [Comment Edited] (HADOOP-16416) mark DynamoDBMetadataStore.deleteTrackingValueMap as final

2019-08-15 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908006#comment-16908006 ] Gabor Bota edited comment on HADOOP-16416 at 8/15/19 11:28 AM: --- You

[jira] [Commented] (HADOOP-16416) mark DynamoDBMetadataStore.deleteTrackingValueMap as final

2019-08-15 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908006#comment-16908006 ] Gabor Bota commented on HADOOP-16416: - +1 on patch v003, even if checkstyle has it's own issues

[jira] [Commented] (HADOOP-16505) Add ability to register custom signer with AWS SignerFactory

2019-08-14 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907281#comment-16907281 ] Gabor Bota commented on HADOOP-16505: - Thanks for working on this [~viczsaurav], and [~jojochuang]

[jira] [Commented] (HADOOP-16416) mark DynamoDBMetadataStore.deleteTrackingValueMap as final

2019-08-14 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907171#comment-16907171 ] Gabor Bota commented on HADOOP-16416: - Thanks for working on this [~pingsutw]! Please note that we

[jira] [Resolved] (HADOOP-16500) S3ADelegationTokens to only log at debug on startup

2019-08-14 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16500. - Resolution: Fixed Fix Version/s: 3.3.0 Committed to trunk. > S3ADelegationTokens to

[jira] [Commented] (HADOOP-16500) S3ADelegationTokens to only log at debug on startup

2019-08-14 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907034#comment-16907034 ] Gabor Bota commented on HADOOP-16500: - +1 for GitHub Pull Request #1269. Committing this. >

[jira] [Created] (HADOOP-16507) S3Guard fsck: Add option to configure severity (level) for the scan

2019-08-12 Thread Gabor Bota (JIRA)
Gabor Bota created HADOOP-16507: --- Summary: S3Guard fsck: Add option to configure severity (level) for the scan Key: HADOOP-16507 URL: https://issues.apache.org/jira/browse/HADOOP-16507 Project: Hadoop

[jira] [Commented] (HADOOP-16138) hadoop fs mkdir / of nonexistent abfs container raises NPE

2019-08-12 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905158#comment-16905158 ] Gabor Bota commented on HADOOP-16138: - Not recently, but I plan to fix this this week. > hadoop fs

[jira] [Commented] (HADOOP-16481) ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region

2019-08-09 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903982#comment-16903982 ] Gabor Bota commented on HADOOP-16481: - thanks for working on this [~ste...@apache.org]; +1;

[jira] [Resolved] (HADOOP-16481) ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region

2019-08-09 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16481. - Resolution: Fixed Fix Version/s: 3.3.0 Target Version/s: 3.3.0 >

[jira] [Resolved] (HADOOP-16499) S3A retry policy to be exponential

2019-08-09 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16499. - Resolution: Fixed Fix Version/s: 3.3.0 Target Version/s: 3.3.0 > S3A retry

[jira] [Commented] (HADOOP-16499) S3A retry policy to be exponential

2019-08-09 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903905#comment-16903905 ] Gabor Bota commented on HADOOP-16499: - +1 on GitHub Pull Request #1246; committing. > S3A retry

[jira] [Created] (HADOOP-16502) Add fsck to S3A tests where additional diagnosis is needed

2019-08-09 Thread Gabor Bota (JIRA)
Gabor Bota created HADOOP-16502: --- Summary: Add fsck to S3A tests where additional diagnosis is needed Key: HADOOP-16502 URL: https://issues.apache.org/jira/browse/HADOOP-16502 Project: Hadoop Common

[jira] [Comment Edited] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-08-06 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900897#comment-16900897 ] Gabor Bota edited comment on HADOOP-16423 at 8/6/19 11:10 AM: -- Based on an

[jira] [Comment Edited] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-08-06 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900897#comment-16900897 ] Gabor Bota edited comment on HADOOP-16423 at 8/6/19 11:06 AM: -- Based on an

[jira] [Comment Edited] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-08-06 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900897#comment-16900897 ] Gabor Bota edited comment on HADOOP-16423 at 8/6/19 11:06 AM: -- Based on an

[jira] [Comment Edited] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-08-06 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900897#comment-16900897 ] Gabor Bota edited comment on HADOOP-16423 at 8/6/19 11:03 AM: -- Based on an

[jira] [Commented] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-08-06 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900897#comment-16900897 ] Gabor Bota commented on HADOOP-16423: - Based on an offline discussion what's coming up: * Severity

[jira] [Comment Edited] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.

2019-08-05 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1697#comment-1697 ] Gabor Bota edited comment on HADOOP-15565 at 8/5/19 11:21 AM: -- Thanks for

[jira] [Commented] (HADOOP-15565) ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.

2019-08-05 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1697#comment-1697 ] Gabor Bota commented on HADOOP-15565: - Thanks for working on this [~LiJinglun]! > I think we

[jira] [Updated] (HADOOP-16483) S3Guarld fsck: Check metadata consistency from metadatastore to S3 (log)

2019-08-02 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16483: Component/s: fs/s3 > S3Guarld fsck: Check metadata consistency from metadatastore to S3 (log) >

[jira] [Updated] (HADOOP-16483) S3Guarld fsck: Check metadata consistency from metadatastore to S3 (log)

2019-08-02 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16483: Affects Version/s: 3.3.0 > S3Guarld fsck: Check metadata consistency from metadatastore to S3

[jira] [Created] (HADOOP-16483) S3Guarld fsck: Check metadata consistency from metadatastore to S3 (log)

2019-08-02 Thread Gabor Bota (JIRA)
Gabor Bota created HADOOP-16483: --- Summary: S3Guarld fsck: Check metadata consistency from metadatastore to S3 (log) Key: HADOOP-16483 URL: https://issues.apache.org/jira/browse/HADOOP-16483 Project:

[jira] [Updated] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-08-02 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16423: Description: This part is only for logging the inconsistencies. This issue only covers the part

[jira] [Updated] (HADOOP-16423) S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log)

2019-08-02 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16423: Summary: S3Guarld fsck: Check metadata consistency from S3 to metadatastore (log) (was:

[jira] [Commented] (HADOOP-16424) S3Guard fsck: Check internal consistency of the MetadataStore

2019-08-02 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898870#comment-16898870 ] Gabor Bota commented on HADOOP-16424: - Tasks to do here: * find orphan entries (entries without a

[jira] [Commented] (HADOOP-16476) Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations

2019-07-31 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897270#comment-16897270 ] Gabor Bota commented on HADOOP-16476: - (y) > Intermittent failure of

[jira] [Updated] (HADOOP-16476) Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations

2019-07-31 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16476: Affects Version/s: 3.3.0 > Intermittent failure of

[jira] [Updated] (HADOOP-16476) Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations

2019-07-31 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16476: Component/s: fs/s3 > Intermittent failure of

[jira] [Created] (HADOOP-16476) Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations

2019-07-31 Thread Gabor Bota (JIRA)
Gabor Bota created HADOOP-16476: --- Summary: Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations Key: HADOOP-16476 URL: https://issues.apache.org/jira/browse/HADOOP-16476

[jira] [Commented] (HADOOP-16472) findbugs warning on LocalMetadataStore.ttlTimeProvider sync

2019-07-30 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896261#comment-16896261 ] Gabor Bota commented on HADOOP-16472: - I see that you already have a PR for this so I've unassigned

[jira] [Assigned] (HADOOP-16472) findbugs warning on LocalMetadataStore.ttlTimeProvider sync

2019-07-30 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16472: --- Assignee: Steve Loughran (was: Gabor Bota) > findbugs warning on

[jira] [Assigned] (HADOOP-16472) findbugs warning on LocalMetadataStore.ttlTimeProvider sync

2019-07-30 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota reassigned HADOOP-16472: --- Assignee: Gabor Bota > findbugs warning on LocalMetadataStore.ttlTimeProvider sync >

[jira] [Commented] (HADOOP-16423) S3Guarld fsck: Check metadata consistency between S3 and metadatastore (log)

2019-07-29 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895414#comment-16895414 ] Gabor Bota commented on HADOOP-16423: - With the first few (WIP) PRs I will add some tests which are

[jira] [Commented] (HADOOP-16423) S3Guarld fsck: Check metadata consistency between S3 and metadatastore (log)

2019-07-29 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895409#comment-16895409 ] Gabor Bota commented on HADOOP-16423: - To implement: * only check flag will work, no fix flag yet

[jira] [Commented] (HADOOP-16380) S3A tombstones can confuse empty directory status

2019-07-23 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890898#comment-16890898 ] Gabor Bota commented on HADOOP-16380: - Thanks for fixing this [~ste...@apache.org], good catch! +1

[jira] [Created] (HADOOP-16442) Intermittent failure of ITestS3GuardToolDynamoDB#testDynamoDBInitDestroyCycle

2019-07-22 Thread Gabor Bota (JIRA)
Gabor Bota created HADOOP-16442: --- Summary: Intermittent failure of ITestS3GuardToolDynamoDB#testDynamoDBInitDestroyCycle Key: HADOOP-16442 URL: https://issues.apache.org/jira/browse/HADOOP-16442

[jira] [Work started] (HADOOP-16433) S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren

2019-07-22 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-16433 started by Gabor Bota. --- > S3Guard: Filter expired entries and tombstones when listing with >

[jira] [Commented] (HADOOP-16433) S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren

2019-07-19 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888979#comment-16888979 ] Gabor Bota commented on HADOOP-16433: - I'll create a jira on refactoring/moving the TTL handling

[jira] [Commented] (HADOOP-16433) S3Guard: Filter expired entries and tombstones when listing with MetadataStore#listChildren

2019-07-19 Thread Gabor Bota (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888976#comment-16888976 ] Gabor Bota commented on HADOOP-16433: - Fixing this is not as straightforward as it may seem first

  1   2   3   4   5   6   7   8   9   10   >