[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180157#comment-16180157 ] Varada Hemeswari commented on HADOOP-14768: --- Thanks [~tmarquardt] for the review. [~ste...@apache.org], Can we make some progress on this with priority? > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14890) Move up to AWS SDK 1.11.199
[
https://issues.apache.org/jira/browse/HADOOP-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180004#comment-16180004
]
Hudson commented on HADOOP-14890:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12974 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12974/])
HADOOP-14890 Move up to AWS SDK 1.11.199. Contributed by Steve Loughran.
(fabbri: rev a2b31e355a73c37dada15a18a90a690314be7fd3)
* (edit) hadoop-project/pom.xml
> Move up to AWS SDK 1.11.199
> ---
>
> Key: HADOOP-14890
> URL: https://issues.apache.org/jira/browse/HADOOP-14890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HADOOP-14890-001.patch
>
>
> the AWS SDK in Hadoop 3.0.-beta-1 prints a warning whenever you call abort()
> on a stream, which is what we need to do whenever doing long-distance seeks
> in a large file opened with fadvise=normal
> {code}
> 2017-09-20 17:51:50,459 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) -
> 2017-09-20 17:51:50,460 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Starting read() [pos =
> 45603305]
> 2017-09-20 17:51:50,461 [ScalaTest-main-running-S3ASeekReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-09-20 17:51:51,263 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Duration of read() [pos =
> 45603305] = 803,650,637 nS
> {code}
> This goes away if we upgrade to the latest SDK, at least for the
> non-localdynamo bits
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14729) Upgrade JUnit 3 test cases to JUnit 4
[ https://issues.apache.org/jira/browse/HADOOP-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1617#comment-1617 ] Ajay Kumar commented on HADOOP-14729: - [~ajisakaa] ,[~ste...@apache.org] thanks for reviewing tedious patch. [~arpitagarwal], thanks for review and commit. > Upgrade JUnit 3 test cases to JUnit 4 > - > > Key: HADOOP-14729 > URL: https://issues.apache.org/jira/browse/HADOOP-14729 > Project: Hadoop Common > Issue Type: Test >Reporter: Akira Ajisaka >Assignee: Ajay Kumar > Labels: newbie > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14729.001.patch, HADOOP-14729.002.patch, > HADOOP-14729.003.patch, HADOOP-14729.004.patch, HADOOP-14729.005.patch, > HADOOP-14729.006.patch, HADOOP-14729.007.patch, HADOOP-14729.008.patch, > HADOOP-14729.009.patch, HADOOP-14729.010.patch, HADOOP-14729.011.patch, > HADOOP-14729.012.patch > > > There are still test classes that extend from junit.framework.TestCase in > hadoop-common. Upgrade them to JUnit4. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14890) Move up to AWS SDK 1.11.199
[
https://issues.apache.org/jira/browse/HADOOP-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179998#comment-16179998
]
Aaron Fabbri commented on HADOOP-14890:
---
Committed to trunk and branch-3.0. Scale, unit, and integration tests all
passed in us-west-2. Thank you for the patch [~ste...@apache.org]
> Move up to AWS SDK 1.11.199
> ---
>
> Key: HADOOP-14890
> URL: https://issues.apache.org/jira/browse/HADOOP-14890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 2.8.0
>
> Attachments: HADOOP-14890-001.patch
>
>
> the AWS SDK in Hadoop 3.0.-beta-1 prints a warning whenever you call abort()
> on a stream, which is what we need to do whenever doing long-distance seeks
> in a large file opened with fadvise=normal
> {code}
> 2017-09-20 17:51:50,459 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) -
> 2017-09-20 17:51:50,460 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Starting read() [pos =
> 45603305]
> 2017-09-20 17:51:50,461 [ScalaTest-main-running-S3ASeekReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-09-20 17:51:51,263 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Duration of read() [pos =
> 45603305] = 803,650,637 nS
> {code}
> This goes away if we upgrade to the latest SDK, at least for the
> non-localdynamo bits
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14890) Move up to AWS SDK 1.11.199
[
https://issues.apache.org/jira/browse/HADOOP-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-14890:
--
Fix Version/s: (was: 2.8.0)
3.0.0
> Move up to AWS SDK 1.11.199
> ---
>
> Key: HADOOP-14890
> URL: https://issues.apache.org/jira/browse/HADOOP-14890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: HADOOP-14890-001.patch
>
>
> the AWS SDK in Hadoop 3.0.-beta-1 prints a warning whenever you call abort()
> on a stream, which is what we need to do whenever doing long-distance seeks
> in a large file opened with fadvise=normal
> {code}
> 2017-09-20 17:51:50,459 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) -
> 2017-09-20 17:51:50,460 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Starting read() [pos =
> 45603305]
> 2017-09-20 17:51:50,461 [ScalaTest-main-running-S3ASeekReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-09-20 17:51:51,263 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Duration of read() [pos =
> 45603305] = 803,650,637 nS
> {code}
> This goes away if we upgrade to the latest SDK, at least for the
> non-localdynamo bits
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14890) Move up to AWS SDK 1.11.199
[
https://issues.apache.org/jira/browse/HADOOP-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-14890:
--
Release Note: (was: Committed to trunk and branch-3.0. Scale, unit, and
integration tests all passed in us-west-2. Thank you for the patch Steve L.
)
> Move up to AWS SDK 1.11.199
> ---
>
> Key: HADOOP-14890
> URL: https://issues.apache.org/jira/browse/HADOOP-14890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 2.8.0
>
> Attachments: HADOOP-14890-001.patch
>
>
> the AWS SDK in Hadoop 3.0.-beta-1 prints a warning whenever you call abort()
> on a stream, which is what we need to do whenever doing long-distance seeks
> in a large file opened with fadvise=normal
> {code}
> 2017-09-20 17:51:50,459 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) -
> 2017-09-20 17:51:50,460 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Starting read() [pos =
> 45603305]
> 2017-09-20 17:51:50,461 [ScalaTest-main-running-S3ASeekReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-09-20 17:51:51,263 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Duration of read() [pos =
> 45603305] = 803,650,637 nS
> {code}
> This goes away if we upgrade to the latest SDK, at least for the
> non-localdynamo bits
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14890) Move up to AWS SDK 1.11.199
[
https://issues.apache.org/jira/browse/HADOOP-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-14890:
--
Resolution: Fixed
Release Note:
Committed to trunk and branch-3.0. Scale, unit, and integration tests all
passed in us-west-2. Thank you for the patch Steve L.
Status: Resolved (was: Patch Available)
> Move up to AWS SDK 1.11.199
> ---
>
> Key: HADOOP-14890
> URL: https://issues.apache.org/jira/browse/HADOOP-14890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 2.8.0
>
> Attachments: HADOOP-14890-001.patch
>
>
> the AWS SDK in Hadoop 3.0.-beta-1 prints a warning whenever you call abort()
> on a stream, which is what we need to do whenever doing long-distance seeks
> in a large file opened with fadvise=normal
> {code}
> 2017-09-20 17:51:50,459 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) -
> 2017-09-20 17:51:50,460 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Starting read() [pos =
> 45603305]
> 2017-09-20 17:51:50,461 [ScalaTest-main-running-S3ASeekReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-09-20 17:51:51,263 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Duration of read() [pos =
> 45603305] = 803,650,637 nS
> {code}
> This goes away if we upgrade to the latest SDK, at least for the
> non-localdynamo bits
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14890) Move up to AWS SDK 1.11.199
[
https://issues.apache.org/jira/browse/HADOOP-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179944#comment-16179944
]
Aaron Fabbri commented on HADOOP-14890:
---
Testing this now (unit, integration, scale). If it looks good I will commit
it.
> Move up to AWS SDK 1.11.199
> ---
>
> Key: HADOOP-14890
> URL: https://issues.apache.org/jira/browse/HADOOP-14890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Fix For: 2.8.0
>
> Attachments: HADOOP-14890-001.patch
>
>
> the AWS SDK in Hadoop 3.0.-beta-1 prints a warning whenever you call abort()
> on a stream, which is what we need to do whenever doing long-distance seeks
> in a large file opened with fadvise=normal
> {code}
> 2017-09-20 17:51:50,459 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) -
> 2017-09-20 17:51:50,460 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Starting read() [pos =
> 45603305]
> 2017-09-20 17:51:50,461 [ScalaTest-main-running-S3ASeekReadSuite] WARN
> internal.S3AbortableInputStream (S3AbortableInputStream.java:close(163)) -
> Not all bytes were read from the S3ObjectInputStream, aborting HTTP
> connection. This is likely an error and may result in sub-optimal behavior.
> Request only the bytes you need via a ranged GET or drain the input stream
> after use.
> 2017-09-20 17:51:51,263 [ScalaTest-main-running-S3ASeekReadSuite] INFO
> s3.S3ASeekReadSuite (Logging.scala:logInfo(54)) - Duration of read() [pos =
> 45603305] = 803,650,637 nS
> {code}
> This goes away if we upgrade to the latest SDK, at least for the
> non-localdynamo bits
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179940#comment-16179940 ] Hudson commented on HADOOP-14220: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12972 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12972/]) HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity (fabbri: rev 47011d7dd300b0c74bb6cfe25b918c479d718f4f) * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3GuardCLI.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardWriteBack.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolLocal.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/NullMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardListConsistency.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-015.patch, HADOOP-14220-016.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14220: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-015.patch, HADOOP-14220-016.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179924#comment-16179924 ] Aaron Fabbri commented on HADOOP-14220: --- Committed to trunk and branch-3.0. Thanks for the contribution [~ste...@apache.org]! > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-015.patch, HADOOP-14220-016.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179906#comment-16179906
]
Hudson commented on HADOOP-14903:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12971 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12971/])
HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang) (rchiang: rev
cde804b91f2fed514935d186d89ea7cbbf2701f3)
* (edit) hadoop-project/pom.xml
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903.001.patch, HADOOP-14903.002.patch,
> HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14903) Add json-smart explicitly to pom.xml
[
https://issues.apache.org/jira/browse/HADOOP-14903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated HADOOP-14903:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-beta1
Status: Resolved (was: Patch Available)
Committed to trunk and branch-3.0.
Thanks [~andrew.wang] for the review! Thanks [~ste...@apache.org] for finding
this issue originally!
> Add json-smart explicitly to pom.xml
>
>
> Key: HADOOP-14903
> URL: https://issues.apache.org/jira/browse/HADOOP-14903
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
> Fix For: 3.0.0-beta1
>
> Attachments: HADOOP-14903.001.patch, HADOOP-14903.002.patch,
> HADOOP-14903.003.patch
>
>
> With the library update in HADOOP-14799, maven knows how to pull in
> net.minidev:json-smart for tests, but not for packaging. This needs to be
> added to the main project pom in order to avoid this warning:
> {noformat}
> [WARNING] The POM for net.minidev:json-smart:jar:2.3-SNAPSHOT is missing, no
> dependency information available
> {noformat}
> This is pulled in from a few places:
> {noformat}
> [INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.1.0-SNAPSHOT:compile
> [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | | | \- net.minidev:json-smart:jar:2.3:compile
> [INFO] | | \- org.apache.kerby:token-provider:jar:1.0.1:compile
> [INFO] | | \- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
> [INFO] | |+-
> com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
> [INFO] | |\- net.minidev:json-smart:jar:2.3:compile
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179886#comment-16179886 ] Thomas Marquardt commented on HADOOP-14768: --- +1, thanks for removing the risk from the change. If the feedback is good after this is released, we can combine the code paths. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14651) Update okhttp version to 2.7.5
[ https://issues.apache.org/jira/browse/HADOOP-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179847#comment-16179847 ] John Zhuge commented on HADOOP-14651: - Good idea [~ajisakaa]. We should move most libraries towards this model to avoid dependency on multiple versions. > Update okhttp version to 2.7.5 > -- > > Key: HADOOP-14651 > URL: https://issues.apache.org/jira/browse/HADOOP-14651 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang > Attachments: HADOOP-14651.001.patch, HADOOP-14651.002.patch, > HADOOP-14651.003.patch > > > The current artifact is: > com.squareup.okhttp:okhttp:2.4.0 > That version could either be bumped to 2.7.5 (the latest of that line), or > use the latest artifact: > com.squareup.okhttp3:okhttp:3.8.1 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14673) Remove leftover hadoop_xml_escape from functions
[ https://issues.apache.org/jira/browse/HADOOP-14673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179846#comment-16179846 ] Ajay Kumar commented on HADOOP-14673: - [~arpitagarwal],[~aw] Thanks for review and commit. > Remove leftover hadoop_xml_escape from functions > > > Key: HADOOP-14673 > URL: https://issues.apache.org/jira/browse/HADOOP-14673 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Affects Versions: 3.0.0-beta1 >Reporter: Allen Wittenauer >Assignee: Ajay Kumar > Fix For: 3.0.0-beta1 > > Attachments: HADOOP-14673.01.patch > > > This function is no longer needed, so let's purge it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14217) Object Storage: support colon in object path
[ https://issues.apache.org/jira/browse/HADOOP-14217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179821#comment-16179821 ] Yuliya Feldman commented on HADOOP-14217: - Ping [~ste...@apache.org] and others Any more feedback? > Object Storage: support colon in object path > > > Key: HADOOP-14217 > URL: https://issues.apache.org/jira/browse/HADOOP-14217 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/oss >Affects Versions: 2.8.1 >Reporter: Genmao Yu >Assignee: Yuliya Feldman > Attachments: Colon handling in hadoop Path.pdf > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14893) WritableRpcEngine should use Time.monotonicNow
[
https://issues.apache.org/jira/browse/HADOOP-14893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179770#comment-16179770
]
Hadoop QA commented on HADOOP-14893:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 55s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14893 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12888935/HADOOP-14893-2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 415733348f95 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e928ee5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13375/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13375/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13375/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> WritableRpcEngine should use Time.monotonicNow
> --
>
> Key: HADOOP-14893
> URL: https://issues.apache.org/jira/browse/HADOOP-14893
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Chetna Chaudhari
>Assignee:
[jira] [Commented] (HADOOP-14881) LoadGenerator should use Time.monotonicNow() to measure durations
[ https://issues.apache.org/jira/browse/HADOOP-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179752#comment-16179752 ] Hudson commented on HADOOP-14881: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12969 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12969/]) HADOOP-14881. LoadGenerator should use Time.monotonicNow() to measure (jlowe: rev ac05a51bbb2a3fad4e85f9334a3408571967900a) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/loadGenerator/LoadGenerator.java > LoadGenerator should use Time.monotonicNow() to measure durations > - > > Key: HADOOP-14881 > URL: https://issues.apache.org/jira/browse/HADOOP-14881 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Bharat Viswanadham > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14881.01.patch, HADOOP-14881.02.patch, > HADOOP-14881.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14901) ReuseObjectMapper in Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-14901: -- Resolution: Fixed Target Version/s: 2.8.2, 3.1.0 Status: Resolved (was: Patch Available) [~hanishakoneru] Thank you for the contribution. I have committed this patch to branch-2. > ReuseObjectMapper in Hadoop Common > -- > > Key: HADOOP-14901 > URL: https://issues.apache.org/jira/browse/HADOOP-14901 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HADOOP-14901.001.patch, HADOOP-14901-branch-2.001.patch, > HADOOP-14901-branch-2.002.patch > > > It is recommended to reuse ObjectMapper, if possible, for better performance. > We can aslo use ObjectReader or ObjectWriter to replace the ObjectMapper in > some places: they are straightforward and thread safe. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179739#comment-16179739
]
Hadoop QA commented on HADOOP-14220:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14220 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12888943/HADOOP-14220-016.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9e615fcdaa98 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e928ee5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13376/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13376/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch,
> HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch,
> HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch,
>
[jira] [Updated] (HADOOP-14881) LoadGenerator should use Time.monotonicNow() to measure durations
[ https://issues.apache.org/jira/browse/HADOOP-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-14881: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) Thanks, [~bharatviswa]! I committed this to trunk, branch-3.0, and branch-2. > LoadGenerator should use Time.monotonicNow() to measure durations > - > > Key: HADOOP-14881 > URL: https://issues.apache.org/jira/browse/HADOOP-14881 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Bharat Viswanadham > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-14881.01.patch, HADOOP-14881.02.patch, > HADOOP-14881.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14881) LoadGenerator should use Time.monotonicNow() to measure durations
[ https://issues.apache.org/jira/browse/HADOOP-14881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179734#comment-16179734 ] Jason Lowe commented on HADOOP-14881: - +1 for the latest patch. Committing this. > LoadGenerator should use Time.monotonicNow() to measure durations > - > > Key: HADOOP-14881 > URL: https://issues.apache.org/jira/browse/HADOOP-14881 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Bharat Viswanadham > Attachments: HADOOP-14881.01.patch, HADOOP-14881.02.patch, > HADOOP-14881.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14901) ReuseObjectMapper in Hadoop Common
[
https://issues.apache.org/jira/browse/HADOOP-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179717#comment-16179717
]
Hadoop QA commented on HADOOP-14901:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 52 unchanged - 1 fixed = 52 total (was 53) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 47s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
| Timed out junit tests | org.apache.hadoop.log.TestLogLevel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | HADOOP-14901 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12888929/HADOOP-14901-branch-2.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fe1931198d0d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / c0dacda |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13374/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13374/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13374/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ReuseObjectMapper in Hadoop Common
> --
>
> Key: HADOOP-14901
> URL:
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14220: -- Attachment: HADOOP-14220-016.patch Took v15 patch and tested in us-west-2; all clear. Attaching v16 patch which is identical except fixing last checkstyle issue (unused import). Will commit shortly. > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-015.patch, HADOOP-14220-016.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14893) WritableRpcEngine should use Time.monotonicNow
[ https://issues.apache.org/jira/browse/HADOOP-14893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179668#comment-16179668 ] Chetna Chaudhari edited comment on HADOOP-14893 at 9/25/17 7:59 PM: Thanks for reviewing this patch [~ajisakaa]. I have reverted changes made in `WritableRpcInvoker#call` Please review the new patch. was (Author: chetna): Thanks for reviewing this patch [~ajisakaa]. I have reverted changes made in `WritableRpcInvoker#call` . Please review the new patch. > WritableRpcEngine should use Time.monotonicNow > -- > > Key: HADOOP-14893 > URL: https://issues.apache.org/jira/browse/HADOOP-14893 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Chetna Chaudhari >Priority: Minor > Attachments: HADOOP-14893-2.patch, HADOOP-14893.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14893) WritableRpcEngine should use Time.monotonicNow
[ https://issues.apache.org/jira/browse/HADOOP-14893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetna Chaudhari updated HADOOP-14893: -- Attachment: HADOOP-14893-2.patch Thanks for reviewing this patch [~ajisakaa]. I have reverted changes made in `WritableRpcInvoker#call` . Please review the new patch. > WritableRpcEngine should use Time.monotonicNow > -- > > Key: HADOOP-14893 > URL: https://issues.apache.org/jira/browse/HADOOP-14893 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Chetna Chaudhari >Priority: Minor > Attachments: HADOOP-14893-2.patch, HADOOP-14893.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14897) Loosen compatibility guidelines for native dependencies
[ https://issues.apache.org/jira/browse/HADOOP-14897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HADOOP-14897: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 3.0.0-beta1 Status: Resolved (was: Patch Available) Thanks to [~chris.douglas] for filing the issue and the review. Thanks also to [~miklos.szeg...@cloudera.com] and [~aw] for reviews. Committed to trunk and branch-3.0. > Loosen compatibility guidelines for native dependencies > --- > > Key: HADOOP-14897 > URL: https://issues.apache.org/jira/browse/HADOOP-14897 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, native >Reporter: Chris Douglas >Assignee: Daniel Templeton >Priority: Blocker > Fix For: 3.0.0-beta1, 3.1.0 > > Attachments: HADOOP-14897.001.patch, HADOOP-14897.002.patch > > > Within a major version, the compatibility guidelines forbid raising the > minimum required version of any native dependency or tool required to build > native components. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14901) ReuseObjectMapper in Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HADOOP-14901: Attachment: HADOOP-14901-branch-2.002.patch > ReuseObjectMapper in Hadoop Common > -- > > Key: HADOOP-14901 > URL: https://issues.apache.org/jira/browse/HADOOP-14901 > Project: Hadoop Common > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Minor > Attachments: HADOOP-14901.001.patch, HADOOP-14901-branch-2.001.patch, > HADOOP-14901-branch-2.002.patch > > > It is recommended to reuse ObjectMapper, if possible, for better performance. > We can aslo use ObjectReader or ObjectWriter to replace the ObjectMapper in > some places: they are straightforward and thread safe. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179539#comment-16179539 ] Xiao Chen commented on HADOOP-14521: Hi [~shahrs87], I understand this was missed in my initial reviews when it got committed. My apologies for overlooking it, and any frustration it might have brought when reverting this. But we should really fix the incompatible behavior I mentioned above, which patch 11 would address. Could you take a look? > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HADOOP-14521.09.patch, HADOOP-14521.11.patch, > HADOOP-14521-branch-2.8.002.patch, HADOOP-14521-branch-2.8.2.patch, > HADOOP-14521-trunk-10.patch, HDFS-11804-branch-2.8.patch, > HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch, > HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch, > HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[
https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179531#comment-16179531
]
Hadoop QA commented on HADOOP-14220:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 1
new + 21 unchanged - 0 fixed = 22 total (was 21) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14220 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1238/HADOOP-14220-015.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bcd187af165c 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / e928ee5 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13373/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13373/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13373/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> -
>
> Key: HADOOP-14220
> URL: https://issues.apache.org/jira/browse/HADOOP-14220
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
[jira] [Comment Edited] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179511#comment-16179511
]
Georgi Chalakov edited comment on HADOOP-14906 at 9/25/17 6:28 PM:
---
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
{quote}
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{quote}
{code:title=hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
was (Author: georgi):
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
{noformat}
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{noformat}
{code:title=hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14892) MetricsSystemImpl should use Time.monotonicNow for measuring durations
[ https://issues.apache.org/jira/browse/HADOOP-14892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179518#comment-16179518 ] Chetna Chaudhari commented on HADOOP-14892: --- Thanks [~ajisakaa] for reviewing and committing this patch. > MetricsSystemImpl should use Time.monotonicNow for measuring durations > -- > > Key: HADOOP-14892 > URL: https://issues.apache.org/jira/browse/HADOOP-14892 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Chetna Chaudhari >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0 > > Attachments: HADOOP-14892.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179511#comment-16179511
]
Georgi Chalakov edited comment on HADOOP-14906 at 9/25/17 6:27 PM:
---
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{code:title=hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
was (Author: georgi):
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{code:title=hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\AzureNativeFileSystemStore.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179511#comment-16179511
]
Georgi Chalakov edited comment on HADOOP-14906 at 9/25/17 6:27 PM:
---
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
{noformat}
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{noformat}
{code:title=hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
was (Author: georgi):
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{code:title=hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179511#comment-16179511
]
Georgi Chalakov edited comment on HADOOP-14906 at 9/25/17 6:26 PM:
---
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{code:title=hadoop-tools\hadoop-azure\src\main\java\org\apache\hadoop\fs\azure\AzureNativeFileSystemStore.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
was (Author: georgi):
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{code:title=Bar.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179511#comment-16179511
]
Georgi Chalakov edited comment on HADOOP-14906 at 9/25/17 6:26 PM:
---
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
{code:title=Bar.java|borderStyle=solid}
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
{code}
was (Author: georgi):
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179511#comment-16179511
]
Georgi Chalakov commented on HADOOP-14906:
--
I am not sure that this is related to the block compaction change. The debug
message shows no directories in the list for block blobs with compaction. I
posted the code where we check whether the file is in one of those directories
and if it is not we skip BlockBlobAppendStream.
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
if (isBlockBlobWithCompactionKey(key)) {
BlockBlobAppendStream blockBlobOutputStream = new BlockBlobAppendStream(
(CloudBlockBlobWrapper) blob,
keyEncoded,
this.uploadBlockSizeBytes,
true,
getInstrumentedContext());
outputStream = blockBlobOutputStream;
} else {
outputStream = openOutputStream(blob);
}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14220: -- Status: Patch Available (was: Open) > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-015.patch, HADOOP-14220-HADOOP-13345-001.patch, > HADOOP-14220-HADOOP-13345-002.patch, HADOOP-14220-HADOOP-13345-003.patch, > HADOOP-14220-HADOOP-13345-004.patch, HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14651) Update okhttp version to 2.7.5
[ https://issues.apache.org/jira/browse/HADOOP-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179345#comment-16179345 ] Akira Ajisaka commented on HADOOP-14651: Can we set the version of okhttp to a variable in the pom.xml of hadoop-project module and re-use the variable in the dependency? > Update okhttp version to 2.7.5 > -- > > Key: HADOOP-14651 > URL: https://issues.apache.org/jira/browse/HADOOP-14651 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 3.0.0-beta1 >Reporter: Ray Chiang >Assignee: Ray Chiang > Attachments: HADOOP-14651.001.patch, HADOOP-14651.002.patch, > HADOOP-14651.003.patch > > > The current artifact is: > com.squareup.okhttp:okhttp:2.4.0 > That version could either be bumped to 2.7.5 (the latest of that line), or > use the latest artifact: > com.squareup.okhttp3:okhttp:3.8.1 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14892) MetricsSystemImpl should use Time.monotonicNow for measuring durations
[ https://issues.apache.org/jira/browse/HADOOP-14892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179338#comment-16179338 ] Hudson commented on HADOOP-14892: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12966 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12966/]) HADOOP-14892. MetricsSystemImpl should use Time.monotonicNow for (aajisaka: rev 3a10367a17b99e2c3035ec54048f94f334d93227) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java > MetricsSystemImpl should use Time.monotonicNow for measuring durations > -- > > Key: HADOOP-14892 > URL: https://issues.apache.org/jira/browse/HADOOP-14892 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Chetna Chaudhari >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0 > > Attachments: HADOOP-14892.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14892) MetricsSystemImpl should use Time.monotonicNow for measuring durations
[ https://issues.apache.org/jira/browse/HADOOP-14892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14892: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.0 2.8.3 3.0.0-beta1 2.9.0 Status: Resolved (was: Patch Available) +1, committed to trunk, branch-3.0, branch-2, and branch-2.8. Thanks [~chetna] for the contribution. > MetricsSystemImpl should use Time.monotonicNow for measuring durations > -- > > Key: HADOOP-14892 > URL: https://issues.apache.org/jira/browse/HADOOP-14892 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Chetna Chaudhari >Assignee: Chetna Chaudhari >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1, 2.8.3, 3.1.0 > > Attachments: HADOOP-14892.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14893) WritableRpcEngine should use Time.monotonicNow
[
https://issues.apache.org/jira/browse/HADOOP-14893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179294#comment-16179294
]
Akira Ajisaka commented on HADOOP-14893:
Thanks [~chetna] for the patch.
{code}
// Invoke the protocol method
long startTime = Time.monotonicNow();
int qTime = (int) (startTime-receivedTime);
{code}
{{receivedTime}} is from {{Call.timestamp}} and {{Time.now()}} is used for
initializing the variable. We cannot compare {{Time.now()}} and
{{Time.monotonicNow()}}, so would you undo the change in
{{WritableRpcInvoker#call}}?
> WritableRpcEngine should use Time.monotonicNow
> --
>
> Key: HADOOP-14893
> URL: https://issues.apache.org/jira/browse/HADOOP-14893
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Chetna Chaudhari
>Assignee: Chetna Chaudhari
>Priority: Minor
> Attachments: HADOOP-14893.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14870) backport HADOOP-14553 parallel tests to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179156#comment-16179156
]
Hadoop QA commented on HADOOP-14870:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 101 new or modified
test files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
51s{color} | {color:green} root generated 0 new + 1428 unchanged - 2 fixed =
1428 total (was 1430) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 35s{color} | {color:orange} root: The patch generated 121 new + 183
unchanged - 170 fixed = 304 total (was 353) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
24s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
5s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | HADOOP-14870 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1268/HADOOP-14870-branch-2-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 55a656ca87be 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / de5744c |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13372/artifact/patchprocess/diff-checkstyle-root.txt
|
| whitespace |
[jira] [Commented] (HADOOP-14897) Loosen compatibility guidelines for native dependencies
[ https://issues.apache.org/jira/browse/HADOOP-14897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179108#comment-16179108 ] Hudson commented on HADOOP-14897: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12964 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12964/]) HADOOP-14897. Loosen compatibility guidelines for native dependencies (templedf: rev 0807470808156af13e0992285cac070052d86c26) * (edit) hadoop-common-project/hadoop-common/src/site/markdown/Compatibility.md > Loosen compatibility guidelines for native dependencies > --- > > Key: HADOOP-14897 > URL: https://issues.apache.org/jira/browse/HADOOP-14897 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, native >Reporter: Chris Douglas >Assignee: Daniel Templeton >Priority: Blocker > Attachments: HADOOP-14897.001.patch, HADOOP-14897.002.patch > > > Within a major version, the compatibility guidelines forbid raising the > minimum required version of any native dependency or tool required to build > native components. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14870) backport HADOOP-14553 parallel tests to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14870: Status: Patch Available (was: Open) > backport HADOOP-14553 parallel tests to branch-2 > > > Key: HADOOP-14870 > URL: https://issues.apache.org/jira/browse/HADOOP-14870 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14870-branch-2-001.patch, > HADOOP-14870-branch-2-002.patch > > > Backport the HADOOP-14553 parallel test running from trunk to branch-2. > There's some complexity related to The FS Contract base test being JUnit4 in > branch -2, so its not a simple cherrypick. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14870) backport HADOOP-14553 parallel tests to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14870:
Attachment: HADOOP-14870-branch-2-002.patch
Patch 002; removes the JUnit 4 rules & thread naming from the subclasses of
AbstractFileSystemContract.
testing: All tests passing except for {{ITestAzureConcurrentOutOfBandIo}},
which is covered in HADOOP-14906 & reproduced in trunk, so not a direct result
of this patch
> backport HADOOP-14553 parallel tests to branch-2
>
>
> Key: HADOOP-14870
> URL: https://issues.apache.org/jira/browse/HADOOP-14870
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14870-branch-2-001.patch,
> HADOOP-14870-branch-2-002.patch
>
>
> Backport the HADOOP-14553 parallel test running from trunk to branch-2.
> There's some complexity related to The FS Contract base test being JUnit4 in
> branch -2, so its not a simple cherrypick.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14870) backport HADOOP-14553 parallel tests to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14870: Status: Open (was: Patch Available) > backport HADOOP-14553 parallel tests to branch-2 > > > Key: HADOOP-14870 > URL: https://issues.apache.org/jira/browse/HADOOP-14870 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14870-branch-2-001.patch > > > Backport the HADOOP-14553 parallel test running from trunk to branch-2. > There's some complexity related to The FS Contract base test being JUnit4 in > branch -2, so its not a simple cherrypick. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179043#comment-16179043
]
Steve Loughran commented on HADOOP-14906:
-
Most recent code which has touched this test code (and presumably, the upload
logic) is HADOOP-14520. [~Georgi]: does this stack trace look familiar?
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179039#comment-16179039
]
Steve Loughran commented on HADOOP-14906:
-
And run with azure log at debug
{code}
2017-09-25 14:47:16,936 INFO [JUnit-testReadOOBWrites]: impl.MetricsConfig
(MetricsConfig.java:loadFirst(115)) - loaded properties from
hadoop-metrics2-azure-file-system.properties
2017-09-25 14:47:16,959 INFO [JUnit-testReadOOBWrites]:
impl.MetricsSinkAdapter (MetricsSinkAdapter.java:start(207)) - Sink
azuretestcollector started
2017-09-25 14:47:17,448 INFO [JUnit-testReadOOBWrites]: impl.MetricsSystemImpl
(MetricsSystemImpl.java:startTimer(374)) - Scheduled Metric snapshot period at
10 second(s).
2017-09-25 14:47:17,449 INFO [JUnit-testReadOOBWrites]: impl.MetricsSystemImpl
(MetricsSystemImpl.java:start(191)) - azure-file-system metrics system started
2017-09-25 14:47:17,481 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:configureAzureStorageSession(813)) -
AzureNativeFileSystemStore init.
Settings=8,true,90,{3000,3000,3,30},{true,1.0,1.0}
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(542)) - Page blob directories:
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(550)) - Block blobs with compaction
directories:
2017-09-25 14:47:17,484 DEBUG [JUnit-testReadOOBWrites]:
azure.AzureNativeFileSystemStore
(AzureNativeFileSystemStore.java:initialize(567)) - Atomic rename directories:
/hbase
2017-09-25 14:47:17,501 DEBUG [JUnit-testReadOOBWrites]:
azure.SelfThrottlingIntercept
(SelfThrottlingIntercept.java:sendingRequest(167)) - SelfThrottlingIntercept::
SendingRequest: threadId=11, requestType=read , isFirstRequest=true,
sleepDuration=0
2017-09-25 14:47:17,536 DEBUG [JUnit-testReadOOBWrites]:
azure.SelfThrottlingIntercept
(SelfThrottlingIntercept.java:responseReceived(115)) -
SelfThrottlingIntercept:: ResponseReceived: threadId=11, Status=200,
Elapsed(ms)=34, ETAG="0x8D5041BEF5D2DAF", contentLength=-1, requestMethod=HEAD
2017-09-25 14:47:17,538 DEBUG [JUnit-testReadOOBWrites]:
azure.SelfThrottlingIntercept
(SelfThrottlingIntercept.java:sendingRequest(167)) - SelfThrottlingIntercept::
SendingRequest: threadId=11, requestType=write, isFirstRequest=true,
sleepDuration=0
2017-09-25 14:47:17,569 DEBUG [JUnit-testReadOOBWrites]:
azure.SelfThrottlingIntercept
(SelfThrottlingIntercept.java:responseReceived(115)) -
SelfThrottlingIntercept:: ResponseReceived: threadId=11, Status=200,
Elapsed(ms)=30, ETAG="0x8D5041BEF87544D", contentLength=-1, requestMethod=PUT
2017-09-25 14:47:17,637 DEBUG [pool-1-thread-2]: azure.SelfThrottlingIntercept
(SelfThrottlingIntercept.java:sendingRequest(167)) - SelfThrottlingIntercept::
SendingRequest: threadId=18, requestType=write, isFirstRequest=true,
sleepDuration=0
2017-09-25 14:47:17,637 DEBUG [pool-1-thread-1]: azure.SelfThrottlingIntercept
(SelfThrottlingIntercept.java:sendingRequest(167)) - SelfThrottlingIntercept::
SendingRequest: threadId=17, requestType=write, isFirstRequest=true,
sleepDuration=0
2017-09-25 14:47:22,869 DEBUG [pool-1-thread-2]: azure.SelfThrottlingIntercept
(SelfThrottlingIntercept.java:responseReceived(115)) -
SelfThrottlingIntercept:: ResponseReceived: threadId=18, Status=400,
Elapsed(ms)=5229, ETAG=null, contentLength=405, requestMethod=PUT
2017-09-25 14:47:22,892 INFO [JUnit-testReadOOBWrites]:
azure.AbstractWasbTestBase (AbstractWasbTestBase.java:describe(172)) -
testReadOOBWrites: closing test account and filesystem
java.io.IOException
at
com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:770)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:443)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:52)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:387)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: The MD5 value
specified in the request did not match with the MD5 value calculated by the
server.
at
[jira] [Commented] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179036#comment-16179036
]
Steve Loughran commented on HADOOP-14906:
-
Also surfaces in {{TestAzureConcurrentOutOfBandIoWithSecureMode}}
{code}
testReadOOBWrites(org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode)
Time elapsed: 7.667 sec <<< ERROR!
java.io.IOException: null
at
com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:770)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:443)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:52)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:387)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: The MD5 value
specified in the request did not match with the MD5 value calculated by the
server.
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
at
com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:315)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1078)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:1050)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:437)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:52)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:387)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
[
https://issues.apache.org/jira/browse/HADOOP-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179034#comment-16179034
]
Steve Loughran commented on HADOOP-14906:
-
Stack. The initial exception, "null", is a sign that the Azure SDK isn't
including exception text when it wraps inner exceptions; the nested exception
is reporting a mismatch between the MD5 sent up in a PUT/POST and that received
at the far end.
These tests are being run in a different location/network from usual, if that's
likely to interfere: over HTTPS it shouldn't. Surfaces in branch-2 and trunk.
{code}
testReadOOBWrites(org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo)
Time elapsed: 8.923 sec <<< ERROR!
java.io.IOException: null
at
com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:770)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:443)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:52)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:387)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: The MD5 value
specified in the request did not match with the MD5 value calculated by the
server.
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
at
com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:315)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1078)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:1050)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:437)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:52)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:387)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
> -
>
> Key: HADOOP-14906
> URL: https://issues.apache.org/jira/browse/HADOOP-14906
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Affects Versions: 2.9.0, 3.1.0
> Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
> ireland
>Reporter: Steve Loughran
>
> {{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
> text "The MD5 value specified in the request did not match with the MD5 value
> calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-14845:
---
Assignee: Sivaguru Sankaridurg (was: Steve Loughran)
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14906) ITestAzureConcurrentOutOfBandIo failing with checksum errors on write
Steve Loughran created HADOOP-14906:
---
Summary: ITestAzureConcurrentOutOfBandIo failing with checksum
errors on write
Key: HADOOP-14906
URL: https://issues.apache.org/jira/browse/HADOOP-14906
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/azure
Affects Versions: 2.9.0, 3.1.0
Environment: UK BT ASDL connection, 1.8.0_121-b13, azure storage
ireland
Reporter: Steve Loughran
{{ITestAzureConcurrentOutOfBandIo}} is consistently raising an IOE with the
text "The MD5 value specified in the request did not match with the MD5 value
calculated by the server"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14899) Restrict Access to setPermission operation when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kannapiran Srinivasan updated HADOOP-14899: --- Description: In case of authorization enabled Wasb clusters, we need to restrict setting permissions on files or folders to owner or list of privileged users. Currently in the WASB implementation even when authorization is enabled there is no check happens while doing setPermission call. In this JIRA we would like to add the check on the setPermission call in NativeAzureFileSystem implementation so that only owner or the privileged list of users or daemon users can change the permissions of files/folders was: In case of authorization enabled Wasb clusters, we need to restrict setting permissions on files or folders to owner or list of privileged users. Currently in the WASB implementation even when authorization is enabled there is no check happens while doing setPermission call. In this JIRA we would like to add the check on the setPermission call in NativeAzureFileSystem implementation so that only owner or the privileged list of users can change the permissions of files/folders > Restrict Access to setPermission operation when authorization is enabled in > WASB > > > Key: HADOOP-14899 > URL: https://issues.apache.org/jira/browse/HADOOP-14899 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Kannapiran Srinivasan >Assignee: Kannapiran Srinivasan > Labels: fs, secure, wasb > Attachments: HADOOP-14899-001.patch > > > In case of authorization enabled Wasb clusters, we need to restrict setting > permissions on files or folders to owner or list of privileged users. > Currently in the WASB implementation even when authorization is enabled there > is no check happens while doing setPermission call. In this JIRA we would > like to add the check on the setPermission call in NativeAzureFileSystem > implementation so that only owner or the privileged list of users or daemon > users can change the permissions of files/folders -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14870) backport HADOOP-14553 parallel tests to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178995#comment-16178995
]
Steve Loughran commented on HADOOP-14870:
-
Got the tests working here...they were actually running, though the use of
JUnit4 rules to name threads was amplifying confusion.
Seen a test failure
{code}
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.781 sec <<<
FAILURE! - in org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo
testReadOOBWrites(org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIo)
Time elapsed: 9.78 sec <<< ERROR!
java.io.IOException: null
at
com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:770)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:443)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:52)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:387)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: The MD5 value
specified in the request did not match with the MD5 value calculated by the
server.
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
at
com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:315)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1078)
at
com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:1050)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.writeBlock(BlobOutputStreamInternal.java:437)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal.access$000(BlobOutputStreamInternal.java:52)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:387)
at
com.microsoft.azure.storage.blob.BlobOutputStreamInternal$1.call(BlobOutputStreamInternal.java:384)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Running org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.745 sec <<<
FAILURE! - in
org.apache.hadoop.fs.azure.ITestAzureConcurrentOutOfBandIoWithSecureMode
{code}
> backport HADOOP-14553 parallel tests to branch-2
>
>
> Key: HADOOP-14870
> URL: https://issues.apache.org/jira/browse/HADOOP-14870
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14870-branch-2-001.patch
>
>
> Backport the HADOOP-14553 parallel test running from trunk to branch-2.
> There's some complexity related to The FS Contract base test being JUnit4 in
> branch -2, so its not a simple cherrypick.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14899) Restrict Access to setPermission operation when authorization is enabled in WASB
[
https://issues.apache.org/jira/browse/HADOOP-14899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178967#comment-16178967
]
Hadoop QA commented on HADOOP-14899:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14899 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1245/HADOOP-14899-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 78119dc93598 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d0b2c58 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13371/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13371/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Restrict Access to setPermission operation when authorization is enabled in
> WASB
>
>
> Key: HADOOP-14899
> URL: https://issues.apache.org/jira/browse/HADOOP-14899
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Kannapiran Srinivasan
>Assignee: Kannapiran Srinivasan
> Labels: fs, secure, wasb
> Attachments: HADOOP-14899-001.patch
>
>
> In case of authorization enabled Wasb clusters, we need to restrict setting
> permissions on files or folders to owner or list of
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[
https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178954#comment-16178954
]
Hadoop QA commented on HADOOP-14768:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 2
new + 25 unchanged - 0 fixed = 27 total (was 25) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
59s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14768 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1239/HADOOP-14768.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8e4def465392 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d0b2c58 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13370/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13370/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13370/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Honoring sticky bit during Deletion when authorization is enabled in WASB
> -
>
> Key: HADOOP-14768
> URL: https://issues.apache.org/jira/browse/HADOOP-14768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Varada Hemeswari
>Assignee: Varada Hemeswari
>
[jira] [Updated] (HADOOP-14899) Restrict Access to setPermission operation when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kannapiran Srinivasan updated HADOOP-14899: --- Status: Patch Available (was: Open) > Restrict Access to setPermission operation when authorization is enabled in > WASB > > > Key: HADOOP-14899 > URL: https://issues.apache.org/jira/browse/HADOOP-14899 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Kannapiran Srinivasan >Assignee: Kannapiran Srinivasan > Labels: fs, secure, wasb > Attachments: HADOOP-14899-001.patch > > > In case of authorization enabled Wasb clusters, we need to restrict setting > permissions on files or folders to owner or list of privileged users. > Currently in the WASB implementation even when authorization is enabled there > is no check happens while doing setPermission call. In this JIRA we would > like to add the check on the setPermission call in NativeAzureFileSystem > implementation so that only owner or the privileged list of users can change > the permissions of files/folders -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14899) Restrict Access to setPermission operation when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kannapiran Srinivasan updated HADOOP-14899: --- Attachment: HADOOP-14899-001.patch > Restrict Access to setPermission operation when authorization is enabled in > WASB > > > Key: HADOOP-14899 > URL: https://issues.apache.org/jira/browse/HADOOP-14899 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Kannapiran Srinivasan >Assignee: Kannapiran Srinivasan > Labels: fs, secure, wasb > Attachments: HADOOP-14899-001.patch > > > In case of authorization enabled Wasb clusters, we need to restrict setting > permissions on files or folders to owner or list of privileged users. > Currently in the WASB implementation even when authorization is enabled there > is no check happens while doing setPermission call. In this JIRA we would > like to add the check on the setPermission call in NativeAzureFileSystem > implementation so that only owner or the privileged list of users can change > the permissions of files/folders -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178910#comment-16178910 ] Varada Hemeswari edited comment on HADOOP-14768 at 9/25/17 12:03 PM: - Addressed comments from [~tmarquardt]. Separated delete paths for authorization enabled and disabled flows in [^HADOOP-14768.006.patch] was (Author: vahemesw): addressed comments from [~tmarquardt]. Seperated delete paths for authorization enabled and disabled flows. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.006.patch addressed comments from [~tmarquardt]. Seperated delete paths for authorization enabled and disabled flows. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14220: Attachment: HADOOP-14220-015.patch Patch 015; patch 014 without the checkstyle changes. checkstyle:check is working locally > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-015.patch, HADOOP-14220-HADOOP-13345-001.patch, > HADOOP-14220-HADOOP-13345-002.patch, HADOOP-14220-HADOOP-13345-003.patch, > HADOOP-14220-HADOOP-13345-004.patch, HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
[ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14220: Status: Open (was: Patch Available) > Enhance S3GuardTool with bucket-info and set-capacity commands, tests > - > > Key: HADOOP-14220 > URL: https://issues.apache.org/jira/browse/HADOOP-14220 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, > HADOOP-14220-009.patch, HADOOP-14220-010.patch, HADOOP-14220-012.patch, > HADOOP-14220-013.patch, HADOOP-14220-013.patch, HADOOP-14220-014.patch, > HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch, > HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, > HADOOP-14220-HADOOP-13345-005.patch > > > Add a diagnostics command to s3guard which does whatever we need to diagnose > problems for a specific (named) s3a url. This is something which can be > attached to bug reports as well as used by developers. > * Properties to log (with provenance attribute, which can track bucket > overrides: s3guard metastore setup, autocreate, capacity, > * table present/absent > * # of keys in DDB table for that bucket? > * any other stats? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14900) LambdaTestUtils won't compile with early versions of Java 8
[
https://issues.apache.org/jira/browse/HADOOP-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14900:
Summary: LambdaTestUtils won't compile with early versions of Java 8 (was:
Errors in trunk with early versions of Java 8)
> LambdaTestUtils won't compile with early versions of Java 8
> ---
>
> Key: HADOOP-14900
> URL: https://issues.apache.org/jira/browse/HADOOP-14900
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>
> Just to document the issue in case other developers run into this. Compiling
> trunk with jdk 1.8u05 gives the following errors when compiling:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project hadoop-aws: Compilation failure: Compilation
> failure:
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> {noformat}
> Based on my testing jdk 1.8u92 doesn't produce this error.
> I don't think this issue needs to be fixed in the code, but documenting it in
> JIRA.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14900) Errors in trunk with early versions of Java 8
[
https://issues.apache.org/jira/browse/HADOOP-14900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178877#comment-16178877
]
Steve Loughran commented on HADOOP-14900:
-
Looks like some arg matching/type erasure stuff. It's going to impact anything
downstream trying to compile test code using closures with LTU. I'll change the
text to match
> Errors in trunk with early versions of Java 8
> -
>
> Key: HADOOP-14900
> URL: https://issues.apache.org/jira/browse/HADOOP-14900
> Project: Hadoop Common
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>
> Just to document the issue in case other developers run into this. Compiling
> trunk with jdk 1.8u05 gives the following errors when compiling:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project hadoop-aws: Compilation failure: Compilation
> failure:
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR]
> /root/hadoop/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
> reference to intercept is ambiguous
> [ERROR] both method
> intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable)
> in org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> {noformat}
> Based on my testing jdk 1.8u92 doesn't produce this error.
> I don't think this issue needs to be fixed in the code, but documenting it in
> JIRA.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
