[jira] [Commented] (HADOOP-16248) Fix MutableQuantiles memory leak
[
https://issues.apache.org/jira/browse/HADOOP-16248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840992#comment-16840992
]
Alexis Daboville commented on HADOOP-16248:
---
Great thanks, reattached the patch with the right filename.
> Fix MutableQuantiles memory leak
>
>
> Key: HADOOP-16248
> URL: https://issues.apache.org/jira/browse/HADOOP-16248
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Alexis Daboville
>Priority: Major
> Attachments: HADOOP-16248.00.patch, mutable-quantiles-leak.png,
> mutable-quantiles.patch
>
>
> In some circumstances (high GC, high CPU usage, creating lots of
> S3AFileSystem) it is possible for MutableQuantiles::scheduler [1] to fall
> behind processing tasks that are submitted to it; because tasks are
> submitted on a regular schedule, the unbounded queue backing the
> {{ExecutorService}} might grow to several gigs [2]. By using
> {{scheduleWithFixedDelay}} instead, we ensure that under pressure this leak
> won't
> happen. In order to mitigate the growth, a simple fix [3] is proposed,
> simply replacing {{scheduler.scheduleAtFixedRate}} by
> {{scheduler.scheduleWithFixedDelay}}.
> [1] it is single threaded and shared across all instances of
> {{MutableQuantiles}}:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L66-L68]
> [2] see attached mutable-quantiles-leak.png.
> [3] mutable-quantiles.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16248) Fix MutableQuantiles memory leak
[
https://issues.apache.org/jira/browse/HADOOP-16248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Daboville updated HADOOP-16248:
--
Attachment: HADOOP-16248.00.patch
> Fix MutableQuantiles memory leak
>
>
> Key: HADOOP-16248
> URL: https://issues.apache.org/jira/browse/HADOOP-16248
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Alexis Daboville
>Priority: Major
> Attachments: HADOOP-16248.00.patch, mutable-quantiles-leak.png,
> mutable-quantiles.patch
>
>
> In some circumstances (high GC, high CPU usage, creating lots of
> S3AFileSystem) it is possible for MutableQuantiles::scheduler [1] to fall
> behind processing tasks that are submitted to it; because tasks are
> submitted on a regular schedule, the unbounded queue backing the
> {{ExecutorService}} might grow to several gigs [2]. By using
> {{scheduleWithFixedDelay}} instead, we ensure that under pressure this leak
> won't
> happen. In order to mitigate the growth, a simple fix [3] is proposed,
> simply replacing {{scheduler.scheduleAtFixedRate}} by
> {{scheduler.scheduleWithFixedDelay}}.
> [1] it is single threaded and shared across all instances of
> {{MutableQuantiles}}:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L66-L68]
> [2] see attached mutable-quantiles-leak.png.
> [3] mutable-quantiles.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on a change in pull request #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error
swagle commented on a change in pull request #822: HDDS-1527. HDDS Datanode
start fails due to datanode.id file read error
URL: https://github.com/apache/hadoop/pull/822#discussion_r284537726
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java
##
@@ -221,8 +226,16 @@ public synchronized static DatanodeDetails
readDatanodeDetailsFrom(File path)
try {
return DatanodeIdYaml.readDatanodeIdFile(path);
} catch (IOException e) {
- throw new IOException("Failed to parse DatanodeDetails from "
- + path.getAbsolutePath(), e);
+ LOG.warn("Error loading DatanodeDetails yaml from " +
+ path.getAbsolutePath(), e);
+ // Try to load as protobuf before giving up
+ try (FileInputStream in = new FileInputStream(path)) {
Review comment:
Thanks for the review @xiaoyuyao, will gladly add the test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492910899 > Seems like cache.put is never called except testcases, cache will always be empty. This jira add's Table Cache, this is not integrated to OM Code, in further jira's it will be integrated, and we shall call this API put. Jira description also has the same information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error
xiaoyuyao commented on a change in pull request #822: HDDS-1527. HDDS Datanode
start fails due to datanode.id file read error
URL: https://github.com/apache/hadoop/pull/822#discussion_r284531715
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java
##
@@ -221,8 +226,16 @@ public synchronized static DatanodeDetails
readDatanodeDetailsFrom(File path)
try {
return DatanodeIdYaml.readDatanodeIdFile(path);
} catch (IOException e) {
- throw new IOException("Failed to parse DatanodeDetails from "
- + path.getAbsolutePath(), e);
+ LOG.warn("Error loading DatanodeDetails yaml from " +
+ path.getAbsolutePath(), e);
+ // Try to load as protobuf before giving up
+ try (FileInputStream in = new FileInputStream(path)) {
Review comment:
@swagle , change looks good to me. Can you add a unit test to valid the
upgrade case where the old protobuf is successfully read and rewrite into yaml
format.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15685) Build fails (hadoop pipes) on newer Linux envs (like Fedora 28)
[ https://issues.apache.org/jira/browse/HADOOP-15685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840954#comment-16840954 ] Feilong He edited comment on HADOOP-15685 at 5/16/19 3:51 AM: -- I encountered the same error when I build Hadoop-3.3-SNAPSHOT on Fedora 29. I applied the patch with fixing some code conflicts on latest trunk. Then, the build succeeded. Thanks [~lhofhansl] for your work. It would be awesome if a formal patch is worked out and merged to upstream. was (Author: philohe): I encountered the same error when I build Hadoop-3.3-SNAPSHOT on Fedora 29. I applied the patch with fixing some code conflicts on latest trunk. Then, the build succeeded. Thanks [~lhofhansl] for your this work. It would be awesome that a formal patch is worked out and merged to upstream. > Build fails (hadoop pipes) on newer Linux envs (like Fedora 28) > --- > > Key: HADOOP-15685 > URL: https://issues.apache.org/jira/browse/HADOOP-15685 > Project: Hadoop Common > Issue Type: Improvement > Components: build, tools/pipes >Affects Versions: 3.2.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: 15685-3.2.0.txt, 15685-example.txt > > > The rpc/types.h and similar includes are no longer part of glibc. > Instead tirpc needs to be used now on those systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] chinphing edited a comment on issue #798: HDDS-1499. OzoneManager Cache.
chinphing edited a comment on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492905610 Seems like cache.put is never called except testcases, cache will always be empty. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] chinphing edited a comment on issue #798: HDDS-1499. OzoneManager Cache.
chinphing edited a comment on issue #798: HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#issuecomment-492905610
Seems like addCacheEntry is never called, cache will always be empty.
`@Override
public void addCacheEntry(CacheKey cacheKey,
CacheValue cacheValue) {
// This will override the entry if there is already entry for this key.
cache.put(cacheKey, cacheValue);
}`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] chinphing commented on issue #798: HDDS-1499. OzoneManager Cache.
chinphing commented on issue #798: HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#issuecomment-492905610
Seems like addCacheEntry is never called, cache will always be empty.
`
@Override
public void addCacheEntry(CacheKey cacheKey,
CacheValue cacheValue) {
// This will override the entry if there is already entry for this key.
cache.put(cacheKey, cacheValue);
}
`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #815: HDDS-1522. Provide intellij runConfiguration for Ozone components
xiaoyuyao commented on a change in pull request #815: HDDS-1522. Provide
intellij runConfiguration for Ozone components
URL: https://github.com/apache/hadoop/pull/815#discussion_r284529730
##
File path: hadoop-ozone/dev-support/intellij/install-runconfigs.sh
##
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+SRC_DIR="$SCRIPT_DIR/runConfigurations"
+DEST_DIR="$SCRIPT_DIR/../../../.idea/runConfigurations/"
+#shellcheck disable=SC2010
+ls -1 "$SRC_DIR" | grep -v ozone-site.xml | xargs -n1 -I FILE cp
"$SRC_DIR/FILE" "$DEST_DIR"
Review comment:
Yes, mkdir -p solves the script problem. However, it is not clear how to
import these xml files into intelliJ. Can you provide some pointers on how to
import these xml files and run them inside IntelliJ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15685) Build fails (hadoop pipes) on newer Linux envs (like Fedora 28)
[
https://issues.apache.org/jira/browse/HADOOP-15685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840960#comment-16840960
]
Hadoop QA commented on HADOOP-15685:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
3s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-15685 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15685 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12956098/15685-3.2.0.txt |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16252/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Build fails (hadoop pipes) on newer Linux envs (like Fedora 28)
> ---
>
> Key: HADOOP-15685
> URL: https://issues.apache.org/jira/browse/HADOOP-15685
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build, tools/pipes
>Affects Versions: 3.2.0
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Major
> Attachments: 15685-3.2.0.txt, 15685-example.txt
>
>
> The rpc/types.h and similar includes are no longer part of glibc.
> Instead tirpc needs to be used now on those systems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15685) Build fails (hadoop pipes) on newer Linux envs (like Fedora 28)
[ https://issues.apache.org/jira/browse/HADOOP-15685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840954#comment-16840954 ] Feilong He commented on HADOOP-15685: - I encountered the same error when I build Hadoop-3.3-SNAPSHOT on Fedora 29. I applied the patch with fixing some code conflicts on latest trunk. Then, the build succeeded. Thanks [~lhofhansl] for your this work. It would be awesome that a formal patch is worked out and merged to upstream. > Build fails (hadoop pipes) on newer Linux envs (like Fedora 28) > --- > > Key: HADOOP-15685 > URL: https://issues.apache.org/jira/browse/HADOOP-15685 > Project: Hadoop Common > Issue Type: Improvement > Components: build, tools/pipes >Affects Versions: 3.2.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Attachments: 15685-3.2.0.txt, 15685-example.txt > > > The rpc/types.h and similar includes are no longer part of glibc. > Instead tirpc needs to be used now on those systems. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16268) Allow custom wrapped exception to be thrown by server if RPC call queue is filled up
[
https://issues.apache.org/jira/browse/HADOOP-16268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840907#comment-16840907
]
CR Hota commented on HADOOP-16268:
--
[~elgoiri] [~brahmareddy] [~xkrogen] [~daryn]
Gentle reminder! Could you help share your thoughts?
> Allow custom wrapped exception to be thrown by server if RPC call queue is
> filled up
>
>
> Key: HADOOP-16268
> URL: https://issues.apache.org/jira/browse/HADOOP-16268
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: CR Hota
>Priority: Major
> Attachments: HADOOP-16268.001.patch
>
>
> In the current implementation of callqueue manager,
> "CallQueueOverflowException" exceptions are always wrapping
> "RetriableException". Through configs servers should be allowed to throw
> custom exceptions based on new use cases.
> In CallQueueManager.java for backoff the below is done
> {code:java}
> // ideally this behavior should be controllable too.
> private void throwBackoff() throws IllegalStateException {
> throw CallQueueOverflowException.DISCONNECT;
> }
> {code}
> Since CallQueueOverflowException only wraps RetriableException clients would
> end up hitting the same server for retries. In use cases that router supports
> these overflowed requests could be handled by another router that shares the
> same state thus distributing load across a cluster of routers better. In the
> absence of any custom exception, current behavior should be supported.
> In CallQueueOverflowException class a new Standby exception wrap should be
> created. Something like the below
> {code:java}
>static final CallQueueOverflowException KEEPALIVE =
> new CallQueueOverflowException(
> new RetriableException(TOO_BUSY),
> RpcStatusProto.ERROR);
> static final CallQueueOverflowException DISCONNECT =
> new CallQueueOverflowException(
> new RetriableException(TOO_BUSY + " - disconnecting"),
> RpcStatusProto.FATAL);
> static final CallQueueOverflowException DISCONNECT2 =
> new CallQueueOverflowException(
> new StandbyException(TOO_BUSY + " - disconnecting"),
> RpcStatusProto.FATAL);
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16248) Fix MutableQuantiles memory leak
[
https://issues.apache.org/jira/browse/HADOOP-16248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840893#comment-16840893
]
Bharat Viswanadham edited comment on HADOOP-16248 at 5/16/19 1:06 AM:
--
Hi [~adaboville]
You can create a patch with below format.
<>.<>.patch. In your case, it will be
HADOOP-16248.00.pathc
was (Author: bharatviswa):
Hi [~adaboville]
You can create a patch with below format.
<>.<>.patch
> Fix MutableQuantiles memory leak
>
>
> Key: HADOOP-16248
> URL: https://issues.apache.org/jira/browse/HADOOP-16248
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Alexis Daboville
>Priority: Major
> Attachments: mutable-quantiles-leak.png, mutable-quantiles.patch
>
>
> In some circumstances (high GC, high CPU usage, creating lots of
> S3AFileSystem) it is possible for MutableQuantiles::scheduler [1] to fall
> behind processing tasks that are submitted to it; because tasks are
> submitted on a regular schedule, the unbounded queue backing the
> {{ExecutorService}} might grow to several gigs [2]. By using
> {{scheduleWithFixedDelay}} instead, we ensure that under pressure this leak
> won't
> happen. In order to mitigate the growth, a simple fix [3] is proposed,
> simply replacing {{scheduler.scheduleAtFixedRate}} by
> {{scheduler.scheduleWithFixedDelay}}.
> [1] it is single threaded and shared across all instances of
> {{MutableQuantiles}}:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L66-L68]
> [2] see attached mutable-quantiles-leak.png.
> [3] mutable-quantiles.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16248) Fix MutableQuantiles memory leak
[
https://issues.apache.org/jira/browse/HADOOP-16248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840893#comment-16840893
]
Bharat Viswanadham commented on HADOOP-16248:
-
Hi [~adaboville]
You can create a patch with below format.
<>.<>.patch
> Fix MutableQuantiles memory leak
>
>
> Key: HADOOP-16248
> URL: https://issues.apache.org/jira/browse/HADOOP-16248
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.2
>Reporter: Alexis Daboville
>Priority: Major
> Attachments: mutable-quantiles-leak.png, mutable-quantiles.patch
>
>
> In some circumstances (high GC, high CPU usage, creating lots of
> S3AFileSystem) it is possible for MutableQuantiles::scheduler [1] to fall
> behind processing tasks that are submitted to it; because tasks are
> submitted on a regular schedule, the unbounded queue backing the
> {{ExecutorService}} might grow to several gigs [2]. By using
> {{scheduleWithFixedDelay}} instead, we ensure that under pressure this leak
> won't
> happen. In order to mitigate the growth, a simple fix [3] is proposed,
> simply replacing {{scheduler.scheduleAtFixedRate}} by
> {{scheduler.scheduleWithFixedDelay}}.
> [1] it is single threaded and shared across all instances of
> {{MutableQuantiles}}:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableQuantiles.java#L66-L68]
> [2] see attached mutable-quantiles-leak.png.
> [3] mutable-quantiles.patch
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16247) NPE in FsUrlConnection
[
https://issues.apache.org/jira/browse/HADOOP-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-16247:
Fix Version/s: 3.2.1
> NPE in FsUrlConnection
> --
>
> Key: HADOOP-16247
> URL: https://issues.apache.org/jira/browse/HADOOP-16247
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-16247-001.patch, HADOOP-16247-002.patch,
> HADOOP-16247-003.patch, HADOOP-16247-004.patch, HADOOP-16247-005.patch,
> HADOOP-16247-006.patch, HADOOP-16247-007.patch, HADOOP-16247-008.patch,
> HADOOP-16247-009.patch
>
>
> FsUrlConnection doesn't handle relativePath correctly after the change
> [HADOOP-15217|https://issues.apache.org/jira/browse/HADOOP-15217]
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:385)
> at org.apache.hadoop.fs.Path.isAbsolute(Path.java:395)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
> at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:62)
> at
> org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:71)
> at java.net.URL.openStream(URL.java:1045)
> at UrlProblem.testRelativePath(UrlProblem.java:33)
> at UrlProblem.main(UrlProblem.java:19)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16247) NPE in FsUrlConnection
[
https://issues.apache.org/jira/browse/HADOOP-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840889#comment-16840889
]
Bharat Viswanadham edited comment on HADOOP-16247 at 5/16/19 1:03 AM:
--
Thank You [~kpalanisamy] for the contribution, [~jojochuang] and [~daryn] for
the review.
I have committed this to the trunk, branch-3.1, and branch-3.2.
was (Author: bharatviswa):
Thank You [~kpalanisamy] for the contribution, [~jojochuang] and [~daryn] for
the review.
I have committed this to trunk and branch-3.1.
> NPE in FsUrlConnection
> --
>
> Key: HADOOP-16247
> URL: https://issues.apache.org/jira/browse/HADOOP-16247
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HADOOP-16247-001.patch, HADOOP-16247-002.patch,
> HADOOP-16247-003.patch, HADOOP-16247-004.patch, HADOOP-16247-005.patch,
> HADOOP-16247-006.patch, HADOOP-16247-007.patch, HADOOP-16247-008.patch,
> HADOOP-16247-009.patch
>
>
> FsUrlConnection doesn't handle relativePath correctly after the change
> [HADOOP-15217|https://issues.apache.org/jira/browse/HADOOP-15217]
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:385)
> at org.apache.hadoop.fs.Path.isAbsolute(Path.java:395)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
> at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:62)
> at
> org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:71)
> at java.net.URL.openStream(URL.java:1045)
> at UrlProblem.testRelativePath(UrlProblem.java:33)
> at UrlProblem.main(UrlProblem.java:19)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16247) NPE in FsUrlConnection
[
https://issues.apache.org/jira/browse/HADOOP-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840892#comment-16840892
]
Hudson commented on HADOOP-16247:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16556 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16556/])
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy.
(bharat: rev d4c8858586eeed2820f3ab21da79603b52c64594)
* (add)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFsUrlConnectionPath.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsUrlConnection.java
> NPE in FsUrlConnection
> --
>
> Key: HADOOP-16247
> URL: https://issues.apache.org/jira/browse/HADOOP-16247
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Fix For: 3.3.0, 3.1.3
>
> Attachments: HADOOP-16247-001.patch, HADOOP-16247-002.patch,
> HADOOP-16247-003.patch, HADOOP-16247-004.patch, HADOOP-16247-005.patch,
> HADOOP-16247-006.patch, HADOOP-16247-007.patch, HADOOP-16247-008.patch,
> HADOOP-16247-009.patch
>
>
> FsUrlConnection doesn't handle relativePath correctly after the change
> [HADOOP-15217|https://issues.apache.org/jira/browse/HADOOP-15217]
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:385)
> at org.apache.hadoop.fs.Path.isAbsolute(Path.java:395)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
> at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:62)
> at
> org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:71)
> at java.net.URL.openStream(URL.java:1045)
> at UrlProblem.testRelativePath(UrlProblem.java:33)
> at UrlProblem.main(UrlProblem.java:19)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16247) NPE in FsUrlConnection
[
https://issues.apache.org/jira/browse/HADOOP-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840889#comment-16840889
]
Bharat Viswanadham edited comment on HADOOP-16247 at 5/16/19 12:55 AM:
---
Thank You [~kpalanisamy] for the contribution, [~jojochuang] and [~daryn] for
the review.
I have committed this to trunk and branch-3.1.
was (Author: bharatviswa):
Thank You [~kpalanisamy] for the contribution, [~jojochuang] and [~daryn] for
the review.
I have committed this to trunk.
> NPE in FsUrlConnection
> --
>
> Key: HADOOP-16247
> URL: https://issues.apache.org/jira/browse/HADOOP-16247
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Fix For: 3.3.0, 3.1.3
>
> Attachments: HADOOP-16247-001.patch, HADOOP-16247-002.patch,
> HADOOP-16247-003.patch, HADOOP-16247-004.patch, HADOOP-16247-005.patch,
> HADOOP-16247-006.patch, HADOOP-16247-007.patch, HADOOP-16247-008.patch,
> HADOOP-16247-009.patch
>
>
> FsUrlConnection doesn't handle relativePath correctly after the change
> [HADOOP-15217|https://issues.apache.org/jira/browse/HADOOP-15217]
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:385)
> at org.apache.hadoop.fs.Path.isAbsolute(Path.java:395)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
> at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:62)
> at
> org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:71)
> at java.net.URL.openStream(URL.java:1045)
> at UrlProblem.testRelativePath(UrlProblem.java:33)
> at UrlProblem.main(UrlProblem.java:19)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16247) NPE in FsUrlConnection
[
https://issues.apache.org/jira/browse/HADOOP-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-16247:
Fix Version/s: 3.1.3
> NPE in FsUrlConnection
> --
>
> Key: HADOOP-16247
> URL: https://issues.apache.org/jira/browse/HADOOP-16247
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Fix For: 3.3.0, 3.1.3
>
> Attachments: HADOOP-16247-001.patch, HADOOP-16247-002.patch,
> HADOOP-16247-003.patch, HADOOP-16247-004.patch, HADOOP-16247-005.patch,
> HADOOP-16247-006.patch, HADOOP-16247-007.patch, HADOOP-16247-008.patch,
> HADOOP-16247-009.patch
>
>
> FsUrlConnection doesn't handle relativePath correctly after the change
> [HADOOP-15217|https://issues.apache.org/jira/browse/HADOOP-15217]
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:385)
> at org.apache.hadoop.fs.Path.isAbsolute(Path.java:395)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
> at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:62)
> at
> org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:71)
> at java.net.URL.openStream(URL.java:1045)
> at UrlProblem.testRelativePath(UrlProblem.java:33)
> at UrlProblem.main(UrlProblem.java:19)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16247) NPE in FsUrlConnection
[
https://issues.apache.org/jira/browse/HADOOP-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-16247:
Resolution: Fixed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Thank You [~kpalanisamy] for the contribution, [~jojochuang] and [~daryn] for
the review.
I have committed this to trunk.
> NPE in FsUrlConnection
> --
>
> Key: HADOOP-16247
> URL: https://issues.apache.org/jira/browse/HADOOP-16247
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16247-001.patch, HADOOP-16247-002.patch,
> HADOOP-16247-003.patch, HADOOP-16247-004.patch, HADOOP-16247-005.patch,
> HADOOP-16247-006.patch, HADOOP-16247-007.patch, HADOOP-16247-008.patch,
> HADOOP-16247-009.patch
>
>
> FsUrlConnection doesn't handle relativePath correctly after the change
> [HADOOP-15217|https://issues.apache.org/jira/browse/HADOOP-15217]
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:385)
> at org.apache.hadoop.fs.Path.isAbsolute(Path.java:395)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
> at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:62)
> at
> org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:71)
> at java.net.URL.openStream(URL.java:1045)
> at UrlProblem.testRelativePath(UrlProblem.java:33)
> at UrlProblem.main(UrlProblem.java:19)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16247) NPE in FsUrlConnection
[
https://issues.apache.org/jira/browse/HADOOP-16247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840882#comment-16840882
]
Bharat Viswanadham commented on HADOOP-16247:
-
As no further comments, I will commit this shortly.
> NPE in FsUrlConnection
> --
>
> Key: HADOOP-16247
> URL: https://issues.apache.org/jira/browse/HADOOP-16247
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.1.2
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Major
> Attachments: HADOOP-16247-001.patch, HADOOP-16247-002.patch,
> HADOOP-16247-003.patch, HADOOP-16247-004.patch, HADOOP-16247-005.patch,
> HADOOP-16247-006.patch, HADOOP-16247-007.patch, HADOOP-16247-008.patch,
> HADOOP-16247-009.patch
>
>
> FsUrlConnection doesn't handle relativePath correctly after the change
> [HADOOP-15217|https://issues.apache.org/jira/browse/HADOOP-15217]
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.isUriPathAbsolute(Path.java:385)
> at org.apache.hadoop.fs.Path.isAbsolute(Path.java:395)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:87)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:636)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:146)
> at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:347)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899)
> at org.apache.hadoop.fs.FsUrlConnection.connect(FsUrlConnection.java:62)
> at
> org.apache.hadoop.fs.FsUrlConnection.getInputStream(FsUrlConnection.java:71)
> at java.net.URL.openStream(URL.java:1045)
> at UrlProblem.testRelativePath(UrlProblem.java:33)
> at UrlProblem.main(UrlProblem.java:19)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #782: HDDS-1461. Optimize listStatus api in OzoneFileSystem
xiaoyuyao commented on a change in pull request #782: HDDS-1461. Optimize
listStatus api in OzoneFileSystem
URL: https://github.com/apache/hadoop/pull/782#discussion_r284485092
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1416,44 +1421,47 @@ public void createDirectory(OmKeyArgs args) throws
IOException {
try {
metadataManager.getLock().acquireBucketLock(volumeName, bucketName);
Review comment:
this needs to be moved out of try block.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xiaoyuyao commented on a change in pull request #782: HDDS-1461. Optimize listStatus api in OzoneFileSystem
xiaoyuyao commented on a change in pull request #782: HDDS-1461. Optimize
listStatus api in OzoneFileSystem
URL: https://github.com/apache/hadoop/pull/782#discussion_r284484826
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1355,15 +1360,15 @@ public OzoneFileStatus getFileStatus(OmKeyArgs args)
throws IOException {
String bucketName = args.getBucketName();
String keyName = args.getKeyName();
-metadataManager.getLock().acquireBucketLock(volumeName, bucketName);
try {
+ metadataManager.getLock().acquireBucketLock(volumeName, bucketName);
Review comment:
why do we move the acquireBucketLock inside the try block? The original
pattern seems good to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
[
https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840849#comment-16840849
]
Aaron Fabbri commented on HADOOP-16279:
---
{quote}With HADOOP-15999 we changed a bit how authoritative mode works. In
S3AFileSystem#innerGetFileStatus if allowAuthoritative is false then it will
send a HEAD request{quote}
You might be able to resolve HADOOP-14468 then.
Caught up on that issue and asked a followup question
[there|https://issues.apache.org/jira/browse/HADOOP-15999?focusedCommentId=16840844=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16840844].
Using TTL or expiry to resolve the OOB problem makes sense.
{quote} AF> why we need more prune() functions added to the MS interface
GB> That prune is for removing expired entries from the ddbms. It uses
last_updated for expiry rather than mod_time.
{quote}
I would err on the side of simplicity and deleting code, especially from public
interfaces like MetadataStore. We want it to be easy to implement and
understand. The prune(optional_prefix, age) contract is basically **deletes as
many entries as possible that are older than age**. (This is not a MUST delete
because some MS implementations may decide to keep some state around for
internal reasons, e.g. dynamo's ancestor requirement, or not wanting to prune
directories due to complexity).
I think you can satisfy the prune() contract with only one of the time bases
(mod time or last updated). It seems like an internal implementation detail
that doesn't need to be exposed.
Interesting related thought: Can we claim that last_updated (metastore write
time) >= mod_time? In general, assuming no clock issues or null fields? Seems
like it. (Argument: you cannot write a FileStatus with mod_time in the future
unless your clocks are messed up).
With that property, can't we just simplify prune to always use last_updated?
Or, either way, it is an internal implementation detail?
{quote}AF> smarter logic that allows you set a policy for handling S3 versus MS
conflicts
GB> So basically what you mean is to add a conflict resolution algorithm when
an entry is expired?
{quote}
Not so much when entry is expired, but when data from S3 conflicts with data
from MS. For example, MS has tombstone but S3 says file exists.
Thanks again for the patch and discussion!
> S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
> ---
>
> Key: HADOOP-16279
> URL: https://issues.apache.org/jira/browse/HADOOP-16279
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> In HADOOP-15621 we implemented TTL for Authoritative Directory Listings and
> added {{ExpirableMetadata}}. {{DDBPathMetadata}} extends {{PathMetadata}}
> extends {{ExpirableMetadata}}, so all metadata entries in ddb can expire, but
> the implementation is not done yet.
> To complete this feature the following should be done:
> * Add new tests for metadata entry and tombstone expiry to {{ITestS3GuardTtl}}
> * Implement metadata entry and tombstone expiry
> I would like to start a debate on whether we need to use separate expiry
> times for entries and tombstones. My +1 on not using separate settings - so
> only one config name and value.
>
> Notes:
> * In HADOOP-13649 the metadata TTL is implemented in LocalMetadataStore,
> using an existing feature in guava's cache implementation. Expiry is set with
> {{fs.s3a.s3guard.local.ttl}}.
> * LocalMetadataStore's TTL and this TTL is different. That TTL is using the
> guava cache's internal solution for the TTL of these entries. This is an
> S3AFileSystem level solution in S3Guard, a layer above all metadata store.
> * This is not the same, and not using the [DDB's TTL
> feature|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html].
> We need a different behavior than what ddb promises: [cleaning once a day
> with a background
> job|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html]
> is not usable for this feature - although it can be used as a general
> cleanup solution separately and independently from S3Guard.
> * Use the same ttl for entries and authoritative directory listing
> * All entries can be expired. Then the returned metadata from the MS will be
> null.
> * Add two new methods pruneExpiredTtl() and pruneExpiredTtl(String keyPrefix)
> to MetadataStore interface. These methods will delete all expired metadata
> from the ms.
> * Use last_updated field in ms for both file metadata and authoritative
> directory expiry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [hadoop] hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache.
hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492853658 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 62 | Maven dependency ordering for branch | | +1 | mvninstall | 387 | trunk passed | | +1 | compile | 204 | trunk passed | | +1 | checkstyle | 52 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 811 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 125 | trunk passed | | 0 | spotbugs | 239 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 416 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 392 | the patch passed | | +1 | compile | 193 | the patch passed | | +1 | javac | 193 | the patch passed | | +1 | checkstyle | 55 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 664 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 126 | the patch passed | | +1 | findbugs | 431 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 138 | hadoop-hdds in the patch failed. | | -1 | unit | 1068 | hadoop-ozone in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 5323 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.pipeline.TestPipelineClose | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 97f2fd6b2c5c 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 77170e7 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/testReport/ | | Max. process+thread count | 4479 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/8/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284481714
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskControllerImpl.java
##
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_DEFAULT;
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_KEY;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.inject.Inject;
+
+/**
+ * Implementation of ReconTaskController.
+ */
+public class ReconTaskControllerImpl implements ReconTaskController {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconTaskControllerImpl.class);
+
+ private Map reconDBUpdateTasks;
+ private ExecutorService executorService;
+ private int threadCount = 1;
+ private final Semaphore taskSemaphore = new Semaphore(1);
+ private final ReconOMMetadataManager omMetadataManager;
+ private Map taskFailureCounter = new HashMap<>();
+ private static final int TASK_FAILURE_THRESHOLD = 2;
+ private ReconTaskStatusDao reconTaskStatusDao;
+
+ @Inject
+ public ReconTaskControllerImpl(OzoneConfiguration configuration,
+ ReconOMMetadataManager omMetadataManager,
+ Configuration sqlConfiguration) {
+this.omMetadataManager = omMetadataManager;
+reconDBUpdateTasks = new HashMap<>();
+threadCount = configuration.getInt(OZONE_RECON_TASK_THREAD_COUNT_KEY,
+OZONE_RECON_TASK_THREAD_COUNT_DEFAULT);
+executorService = Executors.newFixedThreadPool(threadCount);
+reconTaskStatusDao = new ReconTaskStatusDao(sqlConfiguration);
+ }
+
+ @Override
+ public void registerTask(ReconDBUpdateTask task) {
+String taskName = task.getTaskName();
+LOG.info("Registered task " + taskName + " with controller.");
+
+// Store task in Task Map.
+reconDBUpdateTasks.put(taskName, task);
+// Store Task in Task failure tracker.
+taskFailureCounter.put(taskName, new AtomicInteger(0));
+// Create DB record for the task.
+ReconTaskStatus reconTaskStatusRecord = new ReconTaskStatus(taskName,
+0L, 0L);
+reconTaskStatusDao.insert(reconTaskStatusRecord);
+ }
+
+ /**
+ * For every registered task, we try process step twice and then reprocess
+ * once (if process failed twice) to absorb the events. If a task has failed
+ * reprocess call more than 2 times across events, it is unregistered
+ * (blacklisted).
+ * @param events set of events
+ * @throws InterruptedException
+ */
+ @Override
+ public void consumeOMEvents(OMUpdateEventBatch events)
+ throws InterruptedException {
+
+
+taskSemaphore.acquire();
+
+try {
+ Collection> tasks = new ArrayList<>();
+ for (Map.Entry taskEntry :
+ reconDBUpdateTasks.entrySet()) {
+ReconDBUpdateTask task = taskEntry.getValue();
+tasks.add(() -> task.process(events));
+ }
+
+ List failedTasks = new ArrayList<>();
+ List> results = executorService.invokeAll(tasks);
+ for (Future f : results) {
+String taskName =
[jira] [Commented] (HADOOP-15999) S3Guard: Better support for out-of-band operations
[
https://issues.apache.org/jira/browse/HADOOP-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840846#comment-16840846
]
Sean Mackrory commented on HADOOP-15999:
{quote} Seems like a common case{quote}
Yeah that's actually the exact case that motivated this ticket, IIRC...
> S3Guard: Better support for out-of-band operations
> --
>
> Key: HADOOP-15999
> URL: https://issues.apache.org/jira/browse/HADOOP-15999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Sean Mackrory
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15999-007.patch, HADOOP-15999.001.patch,
> HADOOP-15999.002.patch, HADOOP-15999.003.patch, HADOOP-15999.004.patch,
> HADOOP-15999.005.patch, HADOOP-15999.006.patch, HADOOP-15999.008.patch,
> HADOOP-15999.009.patch, out-of-band-operations.patch
>
>
> S3Guard was initially done on the premise that a new MetadataStore would be
> the source of truth, and that it wouldn't provide guarantees if updates were
> done without using S3Guard.
> I've been seeing increased demand for better support for scenarios where
> operations are done on the data that can't reasonably be done with S3Guard
> involved. For example:
> * A file is deleted using S3Guard, and replaced by some other tool. S3Guard
> can't tell the difference between the new file and delete / list
> inconsistency and continues to treat the file as deleted.
> * An S3Guard-ed file is overwritten by a longer file by some other tool. When
> reading the file, only the length of the original file is read.
> We could possibly have smarter behavior here by querying both S3 and the
> MetadataStore (even in cases where we may currently only query the
> MetadataStore in getFileStatus) and use whichever one has the higher modified
> time.
> This kills the performance boost we currently get in some workloads with the
> short-circuited getFileStatus, but we could keep it with authoritative mode
> which should give a larger performance boost. At least we'd get more
> correctness without authoritative mode and a clear declaration of when we can
> make the assumptions required to short-circuit the process. If we can't
> consider S3Guard the source of truth, we need to defer to S3 more.
> We'd need to be extra sure of any locality / time zone issues if we start
> relying on mod_time more directly, but currently we're tracking the
> modification time as returned by S3 anyway.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15999) S3Guard: Better support for out-of-band operations
[
https://issues.apache.org/jira/browse/HADOOP-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840846#comment-16840846
]
Sean Mackrory edited comment on HADOOP-15999 at 5/15/19 10:49 PM:
--
{quote} Seems like a common case{quote}
Yeah that's actually the exact case that motivated this ticket in the first
case, IIRC...
was (Author: mackrorysd):
{quote} Seems like a common case{quote}
Yeah that's actually the exact case that motivated this ticket, IIRC...
> S3Guard: Better support for out-of-band operations
> --
>
> Key: HADOOP-15999
> URL: https://issues.apache.org/jira/browse/HADOOP-15999
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Sean Mackrory
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15999-007.patch, HADOOP-15999.001.patch,
> HADOOP-15999.002.patch, HADOOP-15999.003.patch, HADOOP-15999.004.patch,
> HADOOP-15999.005.patch, HADOOP-15999.006.patch, HADOOP-15999.008.patch,
> HADOOP-15999.009.patch, out-of-band-operations.patch
>
>
> S3Guard was initially done on the premise that a new MetadataStore would be
> the source of truth, and that it wouldn't provide guarantees if updates were
> done without using S3Guard.
> I've been seeing increased demand for better support for scenarios where
> operations are done on the data that can't reasonably be done with S3Guard
> involved. For example:
> * A file is deleted using S3Guard, and replaced by some other tool. S3Guard
> can't tell the difference between the new file and delete / list
> inconsistency and continues to treat the file as deleted.
> * An S3Guard-ed file is overwritten by a longer file by some other tool. When
> reading the file, only the length of the original file is read.
> We could possibly have smarter behavior here by querying both S3 and the
> MetadataStore (even in cases where we may currently only query the
> MetadataStore in getFileStatus) and use whichever one has the higher modified
> time.
> This kills the performance boost we currently get in some workloads with the
> short-circuited getFileStatus, but we could keep it with authoritative mode
> which should give a larger performance boost. At least we'd get more
> correctness without authoritative mode and a clear declaration of when we can
> make the assumptions required to short-circuit the process. If we can't
> consider S3Guard the source of truth, we need to defer to S3 more.
> We'd need to be extra sure of any locality / time zone issues if we start
> relying on mod_time more directly, but currently we're tracking the
> modification time as returned by S3 anyway.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15999) S3Guard: Better support for out-of-band operations
[ https://issues.apache.org/jira/browse/HADOOP-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840844#comment-16840844 ] Aaron Fabbri commented on HADOOP-15999: --- What about in-band delete (create tombstone) and then OOB create? Didn't see this covered in the cases here but maybe I missed it. Seems like a common case (OOB process dropping data in bucket). Might want to add a test case and document this in a new JIRA if it is not already covered here. > S3Guard: Better support for out-of-band operations > -- > > Key: HADOOP-15999 > URL: https://issues.apache.org/jira/browse/HADOOP-15999 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Sean Mackrory >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15999-007.patch, HADOOP-15999.001.patch, > HADOOP-15999.002.patch, HADOOP-15999.003.patch, HADOOP-15999.004.patch, > HADOOP-15999.005.patch, HADOOP-15999.006.patch, HADOOP-15999.008.patch, > HADOOP-15999.009.patch, out-of-band-operations.patch > > > S3Guard was initially done on the premise that a new MetadataStore would be > the source of truth, and that it wouldn't provide guarantees if updates were > done without using S3Guard. > I've been seeing increased demand for better support for scenarios where > operations are done on the data that can't reasonably be done with S3Guard > involved. For example: > * A file is deleted using S3Guard, and replaced by some other tool. S3Guard > can't tell the difference between the new file and delete / list > inconsistency and continues to treat the file as deleted. > * An S3Guard-ed file is overwritten by a longer file by some other tool. When > reading the file, only the length of the original file is read. > We could possibly have smarter behavior here by querying both S3 and the > MetadataStore (even in cases where we may currently only query the > MetadataStore in getFileStatus) and use whichever one has the higher modified > time. > This kills the performance boost we currently get in some workloads with the > short-circuited getFileStatus, but we could keep it with authoritative mode > which should give a larger performance boost. At least we'd get more > correctness without authoritative mode and a clear declaration of when we can > make the assumptions required to short-circuit the process. If we can't > consider S3Guard the source of truth, we need to defer to S3 more. > We'd need to be extra sure of any locality / time zone issues if we start > relying on mod_time more directly, but currently we're tracking the > modification time as returned by S3 anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284480734
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskControllerImpl.java
##
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_DEFAULT;
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_KEY;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.inject.Inject;
+
+/**
+ * Implementation of ReconTaskController.
+ */
+public class ReconTaskControllerImpl implements ReconTaskController {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconTaskControllerImpl.class);
+
+ private Map reconDBUpdateTasks;
+ private ExecutorService executorService;
+ private int threadCount = 1;
+ private final Semaphore taskSemaphore = new Semaphore(1);
+ private final ReconOMMetadataManager omMetadataManager;
+ private Map taskFailureCounter = new HashMap<>();
+ private static final int TASK_FAILURE_THRESHOLD = 2;
+ private ReconTaskStatusDao reconTaskStatusDao;
+
+ @Inject
+ public ReconTaskControllerImpl(OzoneConfiguration configuration,
+ ReconOMMetadataManager omMetadataManager,
+ Configuration sqlConfiguration) {
+this.omMetadataManager = omMetadataManager;
+reconDBUpdateTasks = new HashMap<>();
+threadCount = configuration.getInt(OZONE_RECON_TASK_THREAD_COUNT_KEY,
+OZONE_RECON_TASK_THREAD_COUNT_DEFAULT);
+executorService = Executors.newFixedThreadPool(threadCount);
+reconTaskStatusDao = new ReconTaskStatusDao(sqlConfiguration);
+ }
+
+ @Override
+ public void registerTask(ReconDBUpdateTask task) {
+String taskName = task.getTaskName();
+LOG.info("Registered task " + taskName + " with controller.");
+
+// Store task in Task Map.
+reconDBUpdateTasks.put(taskName, task);
+// Store Task in Task failure tracker.
+taskFailureCounter.put(taskName, new AtomicInteger(0));
+// Create DB record for the task.
+ReconTaskStatus reconTaskStatusRecord = new ReconTaskStatus(taskName,
+0L, 0L);
+reconTaskStatusDao.insert(reconTaskStatusRecord);
+ }
+
+ /**
+ * For every registered task, we try process step twice and then reprocess
+ * once (if process failed twice) to absorb the events. If a task has failed
+ * reprocess call more than 2 times across events, it is unregistered
+ * (blacklisted).
+ * @param events set of events
+ * @throws InterruptedException
+ */
+ @Override
+ public void consumeOMEvents(OMUpdateEventBatch events)
+ throws InterruptedException {
+
+
+taskSemaphore.acquire();
+
+try {
+ Collection> tasks = new ArrayList<>();
+ for (Map.Entry taskEntry :
+ reconDBUpdateTasks.entrySet()) {
+ReconDBUpdateTask task = taskEntry.getValue();
+tasks.add(() -> task.process(events));
+ }
+
+ List failedTasks = new ArrayList<>();
+ List> results = executorService.invokeAll(tasks);
+ for (Future f : results) {
+String taskName =
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284480391
##
File path:
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestReconTaskControllerImpl.java
##
@@ -0,0 +1,172 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static org.apache.hadoop.ozone.om.OMConfigKeys.OZONE_OM_DB_DIRS;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.File;
+import java.util.Collections;
+
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.recon.persistence.AbstractSqlDatabaseTest;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.apache.hadoop.ozone.recon.recovery.ReconOmMetadataManagerImpl;
+import org.hadoop.ozone.recon.schema.ReconInternalSchemaDefinition;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+/**
+ * Class used to test ReconTaskControllerImpl.
+ */
+public class TestReconTaskControllerImpl extends AbstractSqlDatabaseTest {
+
+ private ReconTaskController reconTaskController;
+
+ private Configuration sqlConfiguration;
+ @Before
+ public void setUp() throws Exception {
+
+File omDbDir = temporaryFolder.newFolder();
+OzoneConfiguration ozoneConfiguration = new OzoneConfiguration();
+ozoneConfiguration.set(OZONE_OM_DB_DIRS, omDbDir.getAbsolutePath());
+ReconOMMetadataManager omMetadataManager = new ReconOmMetadataManagerImpl(
+ozoneConfiguration);
+
+sqlConfiguration = getInjector()
+.getInstance(Configuration.class);
+
+ReconInternalSchemaDefinition schemaDefinition = getInjector().
+getInstance(ReconInternalSchemaDefinition.class);
+schemaDefinition.initializeSchema();
+
+reconTaskController = new ReconTaskControllerImpl(ozoneConfiguration,
+omMetadataManager, sqlConfiguration);
+ }
+
+ @Test
+ public void testRegisterTask() throws Exception {
+String taskName = "Dummy_" + System.currentTimeMillis();
+DummyReconDBTask dummyReconDBTask =
+new DummyReconDBTask(taskName, DummyReconDBTask.TaskType.ALWAYS_PASS);
+reconTaskController.registerTask(dummyReconDBTask);
+assertTrue(reconTaskController.getRegisteredTasks().size() == 1);
+assertTrue(reconTaskController.getRegisteredTasks()
+.get(dummyReconDBTask.getTaskName()) == dummyReconDBTask);
+ }
+
+ @Test
+ public void testConsumeOMEvents() throws Exception {
+
+ReconDBUpdateTask reconDBUpdateTaskMock = mock(ReconDBUpdateTask.class);
+when(reconDBUpdateTaskMock.getTablesListeningOn()).thenReturn(Collections
+.EMPTY_LIST);
+when(reconDBUpdateTaskMock.getTaskName()).thenReturn("MockTask");
+when(reconDBUpdateTaskMock.process(any(OMUpdateEventBatch.class)))
+.thenReturn(new ImmutablePair<>("MockTask", true));
+reconTaskController.registerTask(reconDBUpdateTaskMock);
+reconTaskController.consumeOMEvents(
+new OMUpdateEventBatch(Collections.emptyList()));
+
+verify(reconDBUpdateTaskMock, times(1))
+.process(any());
+ }
+
+ @Test
+ public void testFailedTaskRetryLogic() throws Exception {
+String taskName = "Dummy_" + System.currentTimeMillis();
+DummyReconDBTask dummyReconDBTask =
+new DummyReconDBTask(taskName, DummyReconDBTask.TaskType.FAIL_ONCE);
+reconTaskController.registerTask(dummyReconDBTask);
+
+
+long currentTime = System.nanoTime();
+OMDBUpdateEvent.EventInfo eventInfoMock = mock(
+
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284479782
##
File path:
hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestContainerKeyMapperTask.java
##
@@ -176,6 +163,130 @@ public void testRun() throws Exception{
keyPrefixesForContainer.get(containerKeyPrefix).intValue());
}
+ @Test
+ public void testProcess() throws IOException {
Review comment:
Can you rename to what is this processing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284474899
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskControllerImpl.java
##
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_DEFAULT;
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_KEY;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.inject.Inject;
+
+/**
+ * Implementation of ReconTaskController.
+ */
+public class ReconTaskControllerImpl implements ReconTaskController {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconTaskControllerImpl.class);
+
+ private Map reconDBUpdateTasks;
+ private ExecutorService executorService;
+ private int threadCount = 1;
+ private final Semaphore taskSemaphore = new Semaphore(1);
+ private final ReconOMMetadataManager omMetadataManager;
+ private Map taskFailureCounter = new HashMap<>();
+ private static final int TASK_FAILURE_THRESHOLD = 2;
+ private ReconTaskStatusDao reconTaskStatusDao;
+
+ @Inject
+ public ReconTaskControllerImpl(OzoneConfiguration configuration,
+ ReconOMMetadataManager omMetadataManager,
+ Configuration sqlConfiguration) {
+this.omMetadataManager = omMetadataManager;
+reconDBUpdateTasks = new HashMap<>();
+threadCount = configuration.getInt(OZONE_RECON_TASK_THREAD_COUNT_KEY,
+OZONE_RECON_TASK_THREAD_COUNT_DEFAULT);
+executorService = Executors.newFixedThreadPool(threadCount);
+reconTaskStatusDao = new ReconTaskStatusDao(sqlConfiguration);
+ }
+
+ @Override
+ public void registerTask(ReconDBUpdateTask task) {
+String taskName = task.getTaskName();
+LOG.info("Registered task " + taskName + " with controller.");
+
+// Store task in Task Map.
+reconDBUpdateTasks.put(taskName, task);
+// Store Task in Task failure tracker.
+taskFailureCounter.put(taskName, new AtomicInteger(0));
+// Create DB record for the task.
+ReconTaskStatus reconTaskStatusRecord = new ReconTaskStatus(taskName,
+0L, 0L);
+reconTaskStatusDao.insert(reconTaskStatusRecord);
+ }
+
+ /**
+ * For every registered task, we try process step twice and then reprocess
+ * once (if process failed twice) to absorb the events. If a task has failed
+ * reprocess call more than 2 times across events, it is unregistered
+ * (blacklisted).
+ * @param events set of events
+ * @throws InterruptedException
+ */
+ @Override
+ public void consumeOMEvents(OMUpdateEventBatch events)
+ throws InterruptedException {
+
+
+taskSemaphore.acquire();
+
+try {
+ Collection> tasks = new ArrayList<>();
+ for (Map.Entry taskEntry :
+ reconDBUpdateTasks.entrySet()) {
+ReconDBUpdateTask task = taskEntry.getValue();
+tasks.add(() -> task.process(events));
+ }
+
+ List failedTasks = new ArrayList<>();
+ List> results = executorService.invokeAll(tasks);
+ for (Future f : results) {
+String taskName =
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284474695
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskControllerImpl.java
##
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_DEFAULT;
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_KEY;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.inject.Inject;
+
+/**
+ * Implementation of ReconTaskController.
+ */
+public class ReconTaskControllerImpl implements ReconTaskController {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconTaskControllerImpl.class);
+
+ private Map reconDBUpdateTasks;
+ private ExecutorService executorService;
+ private int threadCount = 1;
+ private final Semaphore taskSemaphore = new Semaphore(1);
+ private final ReconOMMetadataManager omMetadataManager;
+ private Map taskFailureCounter = new HashMap<>();
+ private static final int TASK_FAILURE_THRESHOLD = 2;
+ private ReconTaskStatusDao reconTaskStatusDao;
+
+ @Inject
+ public ReconTaskControllerImpl(OzoneConfiguration configuration,
+ ReconOMMetadataManager omMetadataManager,
+ Configuration sqlConfiguration) {
+this.omMetadataManager = omMetadataManager;
+reconDBUpdateTasks = new HashMap<>();
+threadCount = configuration.getInt(OZONE_RECON_TASK_THREAD_COUNT_KEY,
+OZONE_RECON_TASK_THREAD_COUNT_DEFAULT);
+executorService = Executors.newFixedThreadPool(threadCount);
+reconTaskStatusDao = new ReconTaskStatusDao(sqlConfiguration);
+ }
+
+ @Override
+ public void registerTask(ReconDBUpdateTask task) {
+String taskName = task.getTaskName();
+LOG.info("Registered task " + taskName + " with controller.");
+
+// Store task in Task Map.
+reconDBUpdateTasks.put(taskName, task);
+// Store Task in Task failure tracker.
+taskFailureCounter.put(taskName, new AtomicInteger(0));
+// Create DB record for the task.
+ReconTaskStatus reconTaskStatusRecord = new ReconTaskStatus(taskName,
+0L, 0L);
+reconTaskStatusDao.insert(reconTaskStatusRecord);
+ }
+
+ /**
+ * For every registered task, we try process step twice and then reprocess
+ * once (if process failed twice) to absorb the events. If a task has failed
+ * reprocess call more than 2 times across events, it is unregistered
+ * (blacklisted).
+ * @param events set of events
+ * @throws InterruptedException
+ */
+ @Override
+ public void consumeOMEvents(OMUpdateEventBatch events)
+ throws InterruptedException {
+
+
+taskSemaphore.acquire();
+
+try {
+ Collection> tasks = new ArrayList<>();
+ for (Map.Entry taskEntry :
+ reconDBUpdateTasks.entrySet()) {
+ReconDBUpdateTask task = taskEntry.getValue();
+tasks.add(() -> task.process(events));
+ }
+
+ List failedTasks = new ArrayList<>();
+ List> results = executorService.invokeAll(tasks);
+ for (Future f : results) {
+String taskName =
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284474114
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ReconTaskControllerImpl.java
##
@@ -0,0 +1,199 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_DEFAULT;
+import static
org.apache.hadoop.ozone.recon.ReconServerConfigKeys.OZONE_RECON_TASK_THREAD_COUNT_KEY;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.Semaphore;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.ozone.recon.recovery.ReconOMMetadataManager;
+import org.hadoop.ozone.recon.schema.tables.daos.ReconTaskStatusDao;
+import org.hadoop.ozone.recon.schema.tables.pojos.ReconTaskStatus;
+import org.jooq.Configuration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.inject.Inject;
+
+/**
+ * Implementation of ReconTaskController.
+ */
+public class ReconTaskControllerImpl implements ReconTaskController {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(ReconTaskControllerImpl.class);
+
+ private Map reconDBUpdateTasks;
+ private ExecutorService executorService;
+ private int threadCount = 1;
+ private final Semaphore taskSemaphore = new Semaphore(1);
+ private final ReconOMMetadataManager omMetadataManager;
+ private Map taskFailureCounter = new HashMap<>();
+ private static final int TASK_FAILURE_THRESHOLD = 2;
+ private ReconTaskStatusDao reconTaskStatusDao;
+
+ @Inject
+ public ReconTaskControllerImpl(OzoneConfiguration configuration,
+ ReconOMMetadataManager omMetadataManager,
+ Configuration sqlConfiguration) {
+this.omMetadataManager = omMetadataManager;
+reconDBUpdateTasks = new HashMap<>();
+threadCount = configuration.getInt(OZONE_RECON_TASK_THREAD_COUNT_KEY,
+OZONE_RECON_TASK_THREAD_COUNT_DEFAULT);
+executorService = Executors.newFixedThreadPool(threadCount);
+reconTaskStatusDao = new ReconTaskStatusDao(sqlConfiguration);
+ }
+
+ @Override
+ public void registerTask(ReconDBUpdateTask task) {
+String taskName = task.getTaskName();
+LOG.info("Registered task " + taskName + " with controller.");
+
+// Store task in Task Map.
+reconDBUpdateTasks.put(taskName, task);
+// Store Task in Task failure tracker.
+taskFailureCounter.put(taskName, new AtomicInteger(0));
+// Create DB record for the task.
+ReconTaskStatus reconTaskStatusRecord = new ReconTaskStatus(taskName,
+0L, 0L);
+reconTaskStatusDao.insert(reconTaskStatusRecord);
+ }
+
+ /**
+ * For every registered task, we try process step twice and then reprocess
+ * once (if process failed twice) to absorb the events. If a task has failed
+ * reprocess call more than 2 times across events, it is unregistered
+ * (blacklisted).
+ * @param events set of events
+ * @throws InterruptedException
+ */
+ @Override
+ public void consumeOMEvents(OMUpdateEventBatch events)
+ throws InterruptedException {
+
+
+taskSemaphore.acquire();
+
+try {
+ Collection> tasks = new ArrayList<>();
+ for (Map.Entry taskEntry :
+ reconDBUpdateTasks.entrySet()) {
+ReconDBUpdateTask task = taskEntry.getValue();
+tasks.add(() -> task.process(events));
+ }
+
+ List failedTasks = new ArrayList<>();
+ List> results = executorService.invokeAll(tasks);
Review comment:
Here we should use 1 thread per task since that is the
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284472229
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java
##
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.BUCKET_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.VOLUME_TABLE;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.utils.db.CodecRegistry;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.WriteBatch;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class used to listen on OM RocksDB updates.
+ */
+public class OMDBUpdatesHandler extends WriteBatch.Handler{
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMDBUpdatesHandler.class);
+
+ private OMMetadataManager omMetadataManager;
+ private Map tablesNames;
+ private CodecRegistry codecRegistry;
+ private List omdbUpdateEvents = new ArrayList<>();
+
+ public OMDBUpdatesHandler(OMMetadataManager omMetadataManager) {
+this.omMetadataManager = omMetadataManager;
+tablesNames = omMetadataManager.getStore().getTableNames();
+codecRegistry = omMetadataManager.getStore().getCodecRegistry();
+ }
+
+ @Override
+ public void put(int cfIndex, byte[] keyBytes, byte[] valueBytes) throws
+ RocksDBException {
+try {
+ String tableName = tablesNames.get(cfIndex);
+ Class keyType = getKeyType(tableName);
+ Class valueType = getValueType(tableName);
+ if (valueType == null) {
+return;
+ }
+ Object key = codecRegistry.asObject(keyBytes, keyType);
+ Object value = codecRegistry.asObject(valueBytes, valueType);
+ OMDBUpdateEvent.OMUpdateEventBuilder builder =
+ new OMDBUpdateEvent.OMUpdateEventBuilder<>();
+ builder.setTable(tableName);
+ builder.setKey(key);
+ builder.setValue(value);
+ builder.setAction(OMDBUpdateEvent.OMDBUpdateAction.PUT);
+ OMDBUpdateEvent putEvent = builder.build();
+ // Temporarily adding to an event buffer for testing. In subsequent
JIRAs,
+ // a Recon side class will be implemented that requests delta updates
+ // from OM and calls on this handler. In that case, we will fill up
+ // this buffer and pass it on to the ReconTaskController which has
+ // tasks waiting on OM events.
+ omdbUpdateEvents.add(putEvent);
+ LOG.info("Generated OM update Event for table : " + putEvent.getTable()
+ + ", Key = " + putEvent.getKey());
+} catch (IOException ioEx) {
+ LOG.error("Exception when reading key : " + ioEx);
+}
+ }
+
+ @Override
+ public void delete(int cfIndex, byte[] keyBytes) throws RocksDBException {
+try {
+ String tableName = tablesNames.get(cfIndex);
+ Class keyType = getKeyType(tableName);
+ Object key = codecRegistry.asObject(keyBytes, keyType);
+ OMDBUpdateEvent.OMUpdateEventBuilder builder =
+ new OMDBUpdateEvent.OMUpdateEventBuilder<>();
+ builder.setTable(tableName);
+ builder.setKey(key);
+ builder.setAction(OMDBUpdateEvent.OMDBUpdateAction.DELETE);
+ OMDBUpdateEvent deleteEvent = builder.build();
+ LOG.info("Generated OM delete Event for table : " +
deleteEvent.getTable()
+ + ", Key = " + deleteEvent.getKey());
+ omdbUpdateEvents.add(deleteEvent);
+} catch (IOException ioEx) {
+ LOG.error("Exception when reading key : " + ioEx);
+}
+ }
+
+ // There are no use cases yet for the remaining methods in Recon. These
+ // will be implemented as and when need arises.
+
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284472028
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java
##
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.BUCKET_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.VOLUME_TABLE;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.utils.db.CodecRegistry;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.WriteBatch;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class used to listen on OM RocksDB updates.
+ */
+public class OMDBUpdatesHandler extends WriteBatch.Handler{
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(OMDBUpdatesHandler.class);
+
+ private OMMetadataManager omMetadataManager;
+ private Map tablesNames;
+ private CodecRegistry codecRegistry;
+ private List omdbUpdateEvents = new ArrayList<>();
+
+ public OMDBUpdatesHandler(OMMetadataManager omMetadataManager) {
+this.omMetadataManager = omMetadataManager;
+tablesNames = omMetadataManager.getStore().getTableNames();
+codecRegistry = omMetadataManager.getStore().getCodecRegistry();
+ }
+
+ @Override
+ public void put(int cfIndex, byte[] keyBytes, byte[] valueBytes) throws
+ RocksDBException {
+try {
+ String tableName = tablesNames.get(cfIndex);
+ Class keyType = getKeyType(tableName);
+ Class valueType = getValueType(tableName);
+ if (valueType == null) {
+return;
+ }
+ Object key = codecRegistry.asObject(keyBytes, keyType);
+ Object value = codecRegistry.asObject(valueBytes, valueType);
+ OMDBUpdateEvent.OMUpdateEventBuilder builder =
+ new OMDBUpdateEvent.OMUpdateEventBuilder<>();
+ builder.setTable(tableName);
+ builder.setKey(key);
+ builder.setValue(value);
+ builder.setAction(OMDBUpdateEvent.OMDBUpdateAction.PUT);
+ OMDBUpdateEvent putEvent = builder.build();
+ // Temporarily adding to an event buffer for testing. In subsequent
JIRAs,
+ // a Recon side class will be implemented that requests delta updates
+ // from OM and calls on this handler. In that case, we will fill up
+ // this buffer and pass it on to the ReconTaskController which has
+ // tasks waiting on OM events.
+ omdbUpdateEvents.add(putEvent);
+ LOG.info("Generated OM update Event for table : " + putEvent.getTable()
+ + ", Key = " + putEvent.getKey());
+} catch (IOException ioEx) {
+ LOG.error("Exception when reading key : " + ioEx);
+}
+ }
+
+ @Override
+ public void delete(int cfIndex, byte[] keyBytes) throws RocksDBException {
+try {
+ String tableName = tablesNames.get(cfIndex);
Review comment:
Looks like a repetitive code, a Builder pattern for EventInfo would be nice.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon
task interface to update internal DB on updates from OM.
URL: https://github.com/apache/hadoop/pull/819#discussion_r284471626
##
File path:
hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OMDBUpdatesHandler.java
##
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.tasks;
+
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.BUCKET_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.OmMetadataManagerImpl.VOLUME_TABLE;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.OmVolumeArgs;
+import org.apache.hadoop.utils.db.CodecRegistry;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.WriteBatch;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Class used to listen on OM RocksDB updates.
+ */
+public class OMDBUpdatesHandler extends WriteBatch.Handler{
Review comment:
missing space
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM.
swagle commented on a change in pull request #819: HDDS-1501 : Create a Recon task interface to update internal DB on updates from OM. URL: https://github.com/apache/hadoop/pull/819#discussion_r284470960 ## File path: hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServerConfigKeys.java ## @@ -112,6 +112,10 @@ public static final String OZONE_RECON_SQL_MAX_IDLE_CONNECTION_TEST_STMT = "ozone.recon.sql.db.conn.idle.test"; + public static final String OZONE_RECON_TASK_THREAD_COUNT_KEY = + "ozone.recon.task.thread.count"; + public static final int OZONE_RECON_TASK_THREAD_COUNT_DEFAULT = 1; Review comment: Why create a fix sized threadpool vs create a thread per task? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284462072
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -71,6 +96,27 @@ public boolean isEmpty() throws IOException {
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
+}
+ }
+} else {
+ return getFromTable(key);
Review comment:
Understood the comment, updated the code to remove getTable in multiple
places.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284461812
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/PartialTableCache.java
##
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+
+
+/**
+ * This is used for the tables where we don't want to cache entire table in
+ * in-memory.
+ */
+@Private
+@Evolving
+public class PartialTableCache
+implements TableCache{
+
+ private final ConcurrentHashMap cache;
+ private final TreeSet> epochEntries;
+ private ExecutorService executorService;
+
+
+
+ public PartialTableCache() {
+cache = new ConcurrentHashMap<>();
+epochEntries = new TreeSet>();
+// Created a singleThreadExecutor, so one cleanup will be running at a
+// time.
+executorService = Executors.newSingleThreadExecutor();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cachekey) {
+return cache.get(cachekey);
+ }
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+CacheValue cacheValue = (CacheValue) cache.get(cacheKey);
+epochEntries.add(new EpochEntry<>(cacheValue.getEpoch(), cacheKey));
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+executorService.submit(() -> evictCache(epoch));
+ }
+
+ @Override
+ public int size() {
+return cache.size();
+ }
+
+ private void evictCache(long epoch) {
Review comment:
Yes Key will be evicted once double buffer flushes to disk.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle commented on issue #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error
swagle commented on issue #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error URL: https://github.com/apache/hadoop/pull/822#issuecomment-492832955 Unit test failures are unrelated. @arp7 Can you review this change, please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
[
https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840799#comment-16840799
]
Aaron Fabbri edited comment on HADOOP-16279 at 5/15/19 9:34 PM:
{quote}with HADOOP-15999 we changed a bit how authoritative mode works.{quote}
Apologies I missed the later work on that issue. Apache Jira was still
configured with my old email address so I was not getting notifications for
months. I'll take a look now.
was (Author: fabbri):
{quote}with HADOOP-15999 we changed a bit how authoritative mode works.\{quote}
Apologies I missed the later work on that issue. Apache Jira was still
configured with my old email address so I was not getting notifications for
months. I'll take a look now.
> S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
> ---
>
> Key: HADOOP-16279
> URL: https://issues.apache.org/jira/browse/HADOOP-16279
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> In HADOOP-15621 we implemented TTL for Authoritative Directory Listings and
> added {{ExpirableMetadata}}. {{DDBPathMetadata}} extends {{PathMetadata}}
> extends {{ExpirableMetadata}}, so all metadata entries in ddb can expire, but
> the implementation is not done yet.
> To complete this feature the following should be done:
> * Add new tests for metadata entry and tombstone expiry to {{ITestS3GuardTtl}}
> * Implement metadata entry and tombstone expiry
> I would like to start a debate on whether we need to use separate expiry
> times for entries and tombstones. My +1 on not using separate settings - so
> only one config name and value.
>
> Notes:
> * In HADOOP-13649 the metadata TTL is implemented in LocalMetadataStore,
> using an existing feature in guava's cache implementation. Expiry is set with
> {{fs.s3a.s3guard.local.ttl}}.
> * LocalMetadataStore's TTL and this TTL is different. That TTL is using the
> guava cache's internal solution for the TTL of these entries. This is an
> S3AFileSystem level solution in S3Guard, a layer above all metadata store.
> * This is not the same, and not using the [DDB's TTL
> feature|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html].
> We need a different behavior than what ddb promises: [cleaning once a day
> with a background
> job|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html]
> is not usable for this feature - although it can be used as a general
> cleanup solution separately and independently from S3Guard.
> * Use the same ttl for entries and authoritative directory listing
> * All entries can be expired. Then the returned metadata from the MS will be
> null.
> * Add two new methods pruneExpiredTtl() and pruneExpiredTtl(String keyPrefix)
> to MetadataStore interface. These methods will delete all expired metadata
> from the ms.
> * Use last_updated field in ms for both file metadata and authoritative
> directory expiry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
[
https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840799#comment-16840799
]
Aaron Fabbri commented on HADOOP-16279:
---
{quote}with HADOOP-15999 we changed a bit how authoritative mode works.\{quote}
Apologies I missed the later work on that issue. Apache Jira was still
configured with my old email address so I was not getting notifications for
months. I'll take a look now.
> S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)
> ---
>
> Key: HADOOP-16279
> URL: https://issues.apache.org/jira/browse/HADOOP-16279
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> In HADOOP-15621 we implemented TTL for Authoritative Directory Listings and
> added {{ExpirableMetadata}}. {{DDBPathMetadata}} extends {{PathMetadata}}
> extends {{ExpirableMetadata}}, so all metadata entries in ddb can expire, but
> the implementation is not done yet.
> To complete this feature the following should be done:
> * Add new tests for metadata entry and tombstone expiry to {{ITestS3GuardTtl}}
> * Implement metadata entry and tombstone expiry
> I would like to start a debate on whether we need to use separate expiry
> times for entries and tombstones. My +1 on not using separate settings - so
> only one config name and value.
>
> Notes:
> * In HADOOP-13649 the metadata TTL is implemented in LocalMetadataStore,
> using an existing feature in guava's cache implementation. Expiry is set with
> {{fs.s3a.s3guard.local.ttl}}.
> * LocalMetadataStore's TTL and this TTL is different. That TTL is using the
> guava cache's internal solution for the TTL of these entries. This is an
> S3AFileSystem level solution in S3Guard, a layer above all metadata store.
> * This is not the same, and not using the [DDB's TTL
> feature|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/TTL.html].
> We need a different behavior than what ddb promises: [cleaning once a day
> with a background
> job|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/howitworks-ttl.html]
> is not usable for this feature - although it can be used as a general
> cleanup solution separately and independently from S3Guard.
> * Use the same ttl for entries and authoritative directory listing
> * All entries can be expired. Then the returned metadata from the MS will be
> null.
> * Add two new methods pruneExpiredTtl() and pruneExpiredTtl(String keyPrefix)
> to MetadataStore interface. These methods will delete all expired metadata
> from the ms.
> * Use last_updated field in ms for both file metadata and authoritative
> directory expiry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492831553 Thank You @arp7 for offline discussion. I have addressed review the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284458211
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -245,42 +247,50 @@ protected DBStoreBuilder
addOMTablesAndCodecs(DBStoreBuilder builder) {
*/
protected void initializeOmTables() throws IOException {
userTable =
-this.store.getTable(USER_TABLE, String.class, VolumeList.class);
+this.store.getTable(USER_TABLE, String.class, VolumeList.class,
Review comment:
Done. Now caller's no need to specify CacheType.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #823: HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
hadoop-yetus commented on issue #823: HADOOP-16315. ABFS: transform full UPN for named user in AclStatus URL: https://github.com/apache/hadoop/pull/823#issuecomment-492830088 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1036 | trunk passed | | +1 | compile | 26 | trunk passed | | +1 | checkstyle | 18 | trunk passed | | +1 | mvnsite | 33 | trunk passed | | +1 | shadedclient | 698 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 19 | trunk passed | | 0 | spotbugs | 50 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 47 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 25 | the patch passed | | +1 | compile | 25 | the patch passed | | +1 | javac | 25 | the patch passed | | -0 | checkstyle | 13 | hadoop-tools/hadoop-azure: The patch generated 4 new + 2 unchanged - 0 fixed = 6 total (was 2) | | +1 | mvnsite | 27 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 718 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 23 | the patch passed | | +1 | findbugs | 57 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 85 | hadoop-azure in the patch passed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 3008 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-823/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/823 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2b29718fdd6c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 77170e7 | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-823/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-823/1/testReport/ | | Max. process+thread count | 446 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-823/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284458025
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
Removed FullCache, and have the only implementation for TableCache i.e
PartialCache.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r284457687 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/FullTableCache.java ## @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.utils.db.cache; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Evolving; + +import java.util.concurrent.ConcurrentHashMap; + +/** + * This is the full table cache, where it uses concurrentHashMap internally, + * and does not do any evict or cleanup. This full table cache need to be + * used by tables where we want to cache the entire table with out any + * cleanup to the cache + * @param + * @param + */ + +@Private +@Evolving +public class FullTableCache
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284457474
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -31,22 +38,40 @@
*/
public class TypedTable implements Table {
- private Table rawTable;
+ private final Table rawTable;
+
+ private final CodecRegistry codecRegistry;
- private CodecRegistry codecRegistry;
+ private final Class keyType;
- private Class keyType;
+ private final Class valueType;
- private Class valueType;
+ private final TableCache, CacheValue> cache;
public TypedTable(
Table rawTable,
CodecRegistry codecRegistry, Class keyType,
Class valueType) {
+this(rawTable, codecRegistry, keyType, valueType,
+null);
+ }
+
+
+ public TypedTable(
+ Table rawTable,
+ CodecRegistry codecRegistry, Class keyType,
+ Class valueType, TableCache.CACHETYPE cachetype) {
this.rawTable = rawTable;
this.codecRegistry = codecRegistry;
this.keyType = keyType;
this.valueType = valueType;
+if (cachetype == TableCache.CACHETYPE.FULLCACHE) {
Review comment:
Thanks, Anu for the comment, removed the cache type.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#discussion_r284457204 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java ## @@ -44,17 +45,20 @@ */ Table getTable(String name) throws IOException; + /** * Gets an existing TableStore with implicit key/value conversion. * * @param name - Name of the TableStore to get * @param keyType * @param valueType + * @param cachetype - Type of cache need to be used for this table. * @return - TableStore. * @throws IOException on Failure */ Table getTable(String name, - Class keyType, Class valueType) throws IOException; + Class keyType, Class valueType, + TableCache.CACHETYPE cachetype) throws IOException; Review comment: Removed the cache type. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error
hadoop-yetus commented on issue #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error URL: https://github.com/apache/hadoop/pull/822#issuecomment-492826896 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 32 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 421 | trunk passed | | +1 | compile | 203 | trunk passed | | +1 | checkstyle | 56 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 826 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 129 | trunk passed | | 0 | spotbugs | 235 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 418 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 396 | the patch passed | | +1 | compile | 210 | the patch passed | | +1 | javac | 210 | the patch passed | | +1 | checkstyle | 61 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 657 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 124 | the patch passed | | +1 | findbugs | 428 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 140 | hadoop-hdds in the patch failed. | | -1 | unit | 1105 | hadoop-ozone in the patch failed. | | +1 | asflicense | 31 | The patch does not generate ASF License warnings. | | | | 5375 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerCommandHandler | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-822/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/822 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 30d6347eeaad 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 77170e7 | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-822/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-822/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-822/1/testReport/ | | Max. process+thread count | 5289 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service U: hadoop-hdds/container-service | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-822/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ opened a new pull request #823: HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
DadanielZ opened a new pull request #823: HADOOP-16315. ABFS: transform full UPN for named user in AclStatus URL: https://github.com/apache/hadoop/pull/823 Fixed identity conversion for AclEntry when calling getAclStatus() All tests passed my US-west account: xns account: Tests run: 41, Failures: 0, Errors: 0, Skipped: 0 Tests run: 393, Failures: 0, Errors: 0, Skipped: 25 Tests run: 190, Failures: 0, Errors: 0, Skipped: 23 non-xns account: Tests run: 41, Failures: 0, Errors: 0, Skipped: 0 Tests run: 393, Failures: 0, Errors: 0, Skipped: 207 Tests run: 190, Failures: 0, Errors: 0, Skipped: 15 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache.
hadoop-yetus commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492800469 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 34 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for branch | | +1 | mvninstall | 392 | trunk passed | | +1 | compile | 201 | trunk passed | | +1 | checkstyle | 52 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 810 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 132 | trunk passed | | 0 | spotbugs | 234 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 415 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 388 | the patch passed | | +1 | compile | 206 | the patch passed | | +1 | javac | 206 | the patch passed | | +1 | checkstyle | 61 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 664 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 128 | the patch passed | | +1 | findbugs | 435 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 147 | hadoop-hdds in the patch failed. | | -1 | unit | 846 | hadoop-ozone in the patch failed. | | +1 | asflicense | 40 | The patch does not generate ASF License warnings. | | | | 5192 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/798 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux cba0149be2ba 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9569015 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/testReport/ | | Max. process+thread count | 5138 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-798/6/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] swagle opened a new pull request #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error
swagle opened a new pull request #822: HDDS-1527. HDDS Datanode start fails due to datanode.id file read error URL: https://github.com/apache/hadoop/pull/822 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #654: HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename
hadoop-yetus commented on issue #654: HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename URL: https://github.com/apache/hadoop/pull/654#issuecomment-492791717 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 22 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 66 | Maven dependency ordering for branch | | +1 | mvninstall | 1144 | trunk passed | | +1 | compile | 1064 | trunk passed | | +1 | checkstyle | 148 | trunk passed | | +1 | mvnsite | 123 | trunk passed | | +1 | shadedclient | 1056 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 93 | trunk passed | | 0 | spotbugs | 63 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 185 | trunk passed | | -0 | patch | 99 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for patch | | +1 | mvninstall | 76 | the patch passed | | +1 | compile | 978 | the patch passed | | +1 | javac | 978 | the patch passed | | -0 | checkstyle | 149 | root: The patch generated 62 new + 85 unchanged - 2 fixed = 147 total (was 87) | | +1 | mvnsite | 121 | the patch passed | | -1 | whitespace | 0 | The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 735 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 32 | hadoop-tools_hadoop-aws generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1) | | -1 | findbugs | 71 | hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | +1 | unit | 537 | hadoop-common in the patch passed. | | +1 | unit | 287 | hadoop-aws in the patch passed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 8782 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-tools/hadoop-aws | | | org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostFirst implements Comparator but not Serializable At PathOrderComparators.java:Serializable At PathOrderComparators.java:[lines 69-89] | | | org.apache.hadoop.fs.s3a.s3guard.PathOrderComparators$TopmostLast implements Comparator but not Serializable At PathOrderComparators.java:Serializable At PathOrderComparators.java:[lines 98-109] | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/654 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 212b4c9f4b9d 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 9569015 | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/diff-checkstyle-root.txt | | whitespace | https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/whitespace-eol.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/artifact/out/new-findbugs-hadoop-tools_hadoop-aws.html | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/testReport/ | | Max. process+thread count | 1489 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-654/24/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For
[GitHub] [hadoop] bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on issue #798: HDDS-1499. OzoneManager Cache. URL: https://github.com/apache/hadoop/pull/798#issuecomment-492770431 Thank You @arp7 for the review. Few of the review comments I have addressed them, for other's provided reply for the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284392124
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * First it will check from cache, if it has entry return the value
+ * otherwise, get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 merged pull request #812: HDDS-1511. Space tracking for Open Containers in HDDS Volumes. Contributed by Supratim Deka
arp7 merged pull request #812: HDDS-1511. Space tracking for Open Containers in HDDS Volumes. Contributed by Supratim Deka URL: https://github.com/apache/hadoop/pull/812 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284365121
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
The logic for get() actually takes care of that, I feel for bucket/volume
table having full cache will be useful as this is used for all the request
validation's. So, if we have in cache this will be a quick operation instead of
disk access.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284364154
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java
##
@@ -245,42 +247,50 @@ protected DBStoreBuilder
addOMTablesAndCodecs(DBStoreBuilder builder) {
*/
protected void initializeOmTables() throws IOException {
userTable =
-this.store.getTable(USER_TABLE, String.class, VolumeList.class);
+this.store.getTable(USER_TABLE, String.class, VolumeList.class,
Review comment:
As for some tables where SCM does not need to have cache, so that is the
reason for introducing CacheType.
And also for some tables like deletedtable, where this is used by background
thread inOM, these does not need cache.
And as for volume/bucket table we plan to have entire table info, and
cleanup for them is no-op, even after flush to db, where as for partial cache
the cache entries will be cleaned up. This is the only difference between full
table cache vs partial table cache.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
arp7 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284363840
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
Okay let's skip the full cache then for now. We can add it later as an
optimization. All caches will be partial, and if an entry is not found in the
cache we always go to RocksDB. It should simplify the patch a bit. What do you
think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284363194
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/TableCache.java
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+/**
+ * Cache used for RocksDB tables.
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public interface TableCache {
+
+ /**
+ * Return the value for the key if it is present, otherwise return null.
+ * @param cacheKey
+ * @return CACHEVALUE
+ */
+ CACHEVALUE get(CACHEKEY cacheKey);
+
+ /**
+ * Add an entry to the cache, if the key already exists it overrides.
+ * @param cacheKey
+ * @param value
+ */
+ void put(CACHEKEY cacheKey, CACHEVALUE value);
+
+ /**
+ * Removes all the entries from the cache which are having epoch value less
+ * than or equal to specified epoch value.
+ * @param epoch
+ */
+ void cleanup(long epoch);
+
+ /**
+ * Return the size of the cache.
+ * @return size
+ */
+ int size();
+
+ /**
+ * Defines type of cache need to be used by OM RocksDB tables.
+ */
+ enum CACHETYPE {
Review comment:
Actually, this cache is used for correctness reasons for further operations
validation and read operation correctness. (As when double buffer comes in to
the place we don't commit to DB immediately)
And also we are cleaning up the cache after flush, so I am not sure what is
the use of loading entries. This cache is mainly for correctness reasons.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284362406
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/FullTableCache.java
##
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Evolving;
+
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * This is the full table cache, where it uses concurrentHashMap internally,
+ * and does not do any evict or cleanup. This full table cache need to be
+ * used by tables where we want to cache the entire table with out any
+ * cleanup to the cache
+ * @param
+ * @param
+ */
+
+@Private
+@Evolving
+public class FullTableCache
+implements TableCache {
+
+ private final ConcurrentHashMap cache;
+
+ public FullTableCache() {
+cache = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public CACHEVALUE get(CACHEKEY cacheKey) {
+return cache.get(cacheKey);
+ }
+
+
+ @Override
+ public void put(CACHEKEY cacheKey, CACHEVALUE value) {
+cache.put(cacheKey, value);
+ }
+
+ @Override
+ public void cleanup(long epoch) {
+// Do nothing
Review comment:
As for full table cache, we never want to cleanup, it is a do nothing
operation.
Added to the java doc.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] arp7 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
arp7 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284362457
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * First it will check from cache, if it has entry return the value
+ * otherwise, get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
Review comment:
Thanks. Also instead of null we can pass in Optional.absent(), same thing
but makes it very clear that the value can be missing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284361950
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/cache/CacheValue.java
##
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.utils.db.cache;
+
+import java.util.Objects;
+
+/**
+ * CacheValue for the RocksDB Table.
+ * @param
+ */
+public class CacheValue {
+
+ private VALUE value;
+ private OperationType lastOperation;
+ // This value is used for evict entries from cache.
+ // This value is set with ratis transaction context log entry index.
+ private long epoch;
+
+ public CacheValue(VALUE value, OperationType lastOperation, long epoch) {
+Objects.requireNonNull(value, "Value Should not be null in CacheValue");
+this.value = value;
+this.lastOperation = lastOperation;
+this.epoch = epoch;
+ }
+
+ public VALUE getValue() {
+return value;
+ }
+
+ public OperationType getLastOperation() {
+return lastOperation;
+ }
+
+ public long getEpoch() {
+return epoch;
+ }
+
+ /**
+ * Last happened Operation.
Review comment:
Removed this lastOperation field.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284361801
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
+ * if the key is not found.
+ *
+ * First it will check from cache, if it has entry return the value
+ * otherwise, get from the RocksDB table.
+ *
+ * @param key metadata key
+ * @return VALUE
+ * @throws IOException
+ */
@Override
public VALUE get(KEY key) throws IOException {
+// Here the metadata lock will guarantee that cache is not updated for same
+// key during get key.
+if (cache != null) {
+ CacheValue cacheValue = cache.get(new CacheKey<>(key));
+ if (cacheValue == null) {
+return getFromTable(key);
+ } else {
+// Doing this because, if the Cache Value Last operation is deleted
+// means it will eventually removed from DB. So, we should return null.
+if (cacheValue.getLastOperation() != CacheValue.OperationType.DELETED)
{
+ return cacheValue.getValue();
+} else {
+ return null;
Review comment:
Yes, we can pass null value to CacheValue actual value. This will work.
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #654: HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename
steveloughran commented on issue #654: HADOOP-15183 S3Guard store becomes inconsistent after partial failure of rename URL: https://github.com/apache/hadoop/pull/654#issuecomment-492739025 just pushed out my addition of a bulk update context all the way through the commit operation; it's not yet wired up for the commit phase and too many needless PUTs are being generated. This does represent my model of how we can have an ongoing rename/commit operation where we know not to create duplicate parents Not incorporated andrew's comments...will do that separately. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16316) S3A delegation tests fail if you set fs.s3a.secret.key
Steve Loughran created HADOOP-16316: --- Summary: S3A delegation tests fail if you set fs.s3a.secret.key Key: HADOOP-16316 URL: https://issues.apache.org/jira/browse/HADOOP-16316 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3, test Affects Versions: 3.3.0 Reporter: Steve Loughran The ITests for Session and Role DTs in S3A set the encryption option (to verify its propagation). But if you have set an encryption key in the config then test setup will fail Fix: when you set the encryption, clear the options for fs.s3a.encryption.key for the dest bucket -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16316) S3A delegation tests fail if you set fs.s3a.secret.key
[
https://issues.apache.org/jira/browse/HADOOP-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840564#comment-16840564
]
Steve Loughran commented on HADOOP-16316:
-
Stack
{code}
[ERROR]
testYarnCredentialPickup(org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem)
Time elapsed: 0.064 s <<< ERROR!
java.io.IOException: AES256 is enabled but an encryption key was set in
fs.s3a.server-side-encryption.key (key of length 75 ending with b)
at
org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1494)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:338)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:488)
at
org.apache.hadoop.fs.contract.AbstractBondedFSContract.init(AbstractBondedFSContract.java:72)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:177)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationInFileystem.setup(ITestSessionDelegationInFileystem.java:169)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem.setup(ITestRoleDelegationInFileystem.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
> S3A delegation tests fail if you set fs.s3a.secret.key
> --
>
> Key: HADOOP-16316
> URL: https://issues.apache.org/jira/browse/HADOOP-16316
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
>
> The ITests for Session and Role DTs in S3A set the encryption option (to
> verify its propagation). But if you have set an encryption key in the config
> then test setup will fail
> Fix: when you set the encryption, clear the options for fs.s3a.encryption.key
> for the dest bucket
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bharatviswa504 commented on a change in pull request #798: HDDS-1499. OzoneManager Cache.
bharatviswa504 commented on a change in pull request #798: HDDS-1499.
OzoneManager Cache.
URL: https://github.com/apache/hadoop/pull/798#discussion_r284350525
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/TypedTable.java
##
@@ -69,8 +94,40 @@ public boolean isEmpty() throws IOException {
return rawTable.isEmpty();
}
+ /**
+ * Returns the value mapped to the given key in byte array or returns null
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] eyanghwx commented on a change in pull request #814: HDDS-1518. Use /etc/ozone for configuration inside docker-compose
eyanghwx commented on a change in pull request #814: HDDS-1518. Use /etc/ozone for configuration inside docker-compose URL: https://github.com/apache/hadoop/pull/814#discussion_r284348832 ## File path: Dockerfile ## @@ -36,7 +36,9 @@ RUN chown hadoop /opt ADD scripts /opt/ ADD scripts/krb5.conf /etc/ RUN yum install -y krb5-workstation - +RUN mkdir -p /etc/hadoop && mkdir -p /var/log/hadoop && chmod 777 /etc/hadoop && chmod 777 /var/log/hadoop +ENV HADOOP_LOG_DIR=/var/log/hadoop +ENV HADOOP_CONF_DIR=/etc/hadoop Review comment: chmod 777 is a security hole. It should have sticky bit set (1777) to ensure that normal users can not delete file written by root or other users. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16314) Make sure all end point URL is covered by the same AuthenticationFilter
[ https://issues.apache.org/jira/browse/HADOOP-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HADOOP-16314: --- Component/s: security > Make sure all end point URL is covered by the same AuthenticationFilter > --- > > Key: HADOOP-16314 > URL: https://issues.apache.org/jira/browse/HADOOP-16314 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Reporter: Eric Yang >Priority: Major > Attachments: Hadoop Web Security.xlsx, scan.txt > > > In the enclosed spreadsheet, it shows the list of web applications deployed > by Hadoop, and filters applied to each entry point. > Hadoop web protocol impersonation has been inconsistent. Most of entry point > do not support ?doAs parameter. This creates problem for secure gateway like > Knox to proxy Hadoop web interface on behave of the end user. When the > receiving end does not check for ?doAs flag, web interface would be accessed > using proxy user credential. This can lead to all kind of security holes > using path traversal to exploit Hadoop. > In HADOOP-16287, ProxyUserAuthenticationFilter is proposed as solution to > solve the web impersonation problem. This task is to track changes required > in Hadoop code base to apply authentication filter globally for each of the > web service port. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16314) Make sure all end point URL is covered by the same AuthenticationFilter
[ https://issues.apache.org/jira/browse/HADOOP-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HADOOP-16314: --- Attachment: scan.txt > Make sure all end point URL is covered by the same AuthenticationFilter > --- > > Key: HADOOP-16314 > URL: https://issues.apache.org/jira/browse/HADOOP-16314 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > Attachments: Hadoop Web Security.xlsx, scan.txt > > > In the enclosed spreadsheet, it shows the list of web applications deployed > by Hadoop, and filters applied to each entry point. > Hadoop web protocol impersonation has been inconsistent. Most of entry point > do not support ?doAs parameter. This creates problem for secure gateway like > Knox to proxy Hadoop web interface on behave of the end user. When the > receiving end does not check for ?doAs flag, web interface would be accessed > using proxy user credential. This can lead to all kind of security holes > using path traversal to exploit Hadoop. > In HADOOP-16287, ProxyUserAuthenticationFilter is proposed as solution to > solve the web impersonation problem. This task is to track changes required > in Hadoop code base to apply authentication filter globally for each of the > web service port. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16314) Make sure all end point URL is covered by the same AuthenticationFilter
[ https://issues.apache.org/jira/browse/HADOOP-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HADOOP-16314: --- Attachment: Hadoop Web Security.xlsx > Make sure all end point URL is covered by the same AuthenticationFilter > --- > > Key: HADOOP-16314 > URL: https://issues.apache.org/jira/browse/HADOOP-16314 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > Attachments: Hadoop Web Security.xlsx > > > In the enclosed spreadsheet, it shows the list of web applications deployed > by Hadoop, and filters applied to each entry point. > Hadoop web protocol impersonation has been inconsistent. Most of entry point > do not support ?doAs parameter. This creates problem for secure gateway like > Knox to proxy Hadoop web interface on behave of the end user. When the > receiving end does not check for ?doAs flag, web interface would be accessed > using proxy user credential. This can lead to all kind of security holes > using path traversal to exploit Hadoop. > In HADOOP-16287, ProxyUserAuthenticationFilter is proposed as solution to > solve the web impersonation problem. This task is to track changes required > in Hadoop code base to apply authentication filter globally for each of the > web service port. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16315) ABFS: transform full UPN for named user in AclStatus
[ https://issues.apache.org/jira/browse/HADOOP-16315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou updated HADOOP-16315: - Affects Version/s: 3.2.0 Target Version/s: 3.2.1 Component/s: fs/azure > ABFS: transform full UPN for named user in AclStatus > > > Key: HADOOP-16315 > URL: https://issues.apache.org/jira/browse/HADOOP-16315 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > > When converting the identity in AclStatus, only "owner" and "owning group" > are transformed. We need to add the conversion for the AclEntry in AclStatus > too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16315) ABFS: transform full UPN for named user in AclStatus
Da Zhou created HADOOP-16315: Summary: ABFS: transform full UPN for named user in AclStatus Key: HADOOP-16315 URL: https://issues.apache.org/jira/browse/HADOOP-16315 Project: Hadoop Common Issue Type: Sub-task Reporter: Da Zhou Assignee: Da Zhou When converting the identity in AclStatus, only "owner" and "owning group" are transformed. We need to add the conversion for the AclEntry in AclStatus too. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16314) Make sure all end point URL is covered by the same AuthenticationFilter
[ https://issues.apache.org/jira/browse/HADOOP-16314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HADOOP-16314: --- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-16095 > Make sure all end point URL is covered by the same AuthenticationFilter > --- > > Key: HADOOP-16314 > URL: https://issues.apache.org/jira/browse/HADOOP-16314 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Eric Yang >Priority: Major > > In the enclosed spreadsheet, it shows the list of web applications deployed > by Hadoop, and filters applied to each entry point. > Hadoop web protocol impersonation has been inconsistent. Most of entry point > do not support ?doAs parameter. This creates problem for secure gateway like > Knox to proxy Hadoop web interface on behave of the end user. When the > receiving end does not check for ?doAs flag, web interface would be accessed > using proxy user credential. This can lead to all kind of security holes > using path traversal to exploit Hadoop. > In HADOOP-16287, ProxyUserAuthenticationFilter is proposed as solution to > solve the web impersonation problem. This task is to track changes required > in Hadoop code base to apply authentication filter globally for each of the > web service port. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16287) KerberosAuthenticationHandler Trusted Proxy Support for Knox
[ https://issues.apache.org/jira/browse/HADOOP-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HADOOP-16287: --- Issue Type: Sub-task (was: New Feature) Parent: HADOOP-16095 > KerberosAuthenticationHandler Trusted Proxy Support for Knox > > > Key: HADOOP-16287 > URL: https://issues.apache.org/jira/browse/HADOOP-16287 > Project: Hadoop Common > Issue Type: Sub-task > Components: auth >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: HADOOP-16287-001.patch, HADOOP-16287-002.patch, > HADOOP-16287-004.patch, HADOOP-16287-005.patch, HADOOP-16287-006.patch, > HADOOP-16287-007.patch, HADOOP-16827-003.patch > > > Knox passes doAs with end user while accessing RM, WebHdfs Rest Api. > Currently KerberosAuthenticationHandler sets the remote user to Knox. Need > Trusted Proxy Support by reading doAs query parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16095) Support impersonation for AuthenticationFilter
[
https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840542#comment-16840542
]
Eric Yang commented on HADOOP-16095:
The patch 004 was the original patch that posted in Hadoop security mailing
list on Feb 11, 2019. This patch covers a new AuthenticationFilter that
enables impersonation at web protocol. It also covers patch to apply
AuthenticationFilter globally to HDFS and YARN applications. The core filter
is refined in HADOOP-16287. The application of the filter is filed as another
issue HADOOP-16314 to ensure all entry points are covered.
> Support impersonation for AuthenticationFilter
> --
>
> Key: HADOOP-16095
> URL: https://issues.apache.org/jira/browse/HADOOP-16095
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16095.004.patch
>
>
> External services or YARN service may need to call into WebHDFS or YARN REST
> API on behave of the user using web protocols. It would be good to support
> impersonation mechanism in AuthenticationFilter or similar extensions. The
> general design is similar to UserGroupInformation.doAs in RPC layer.
> The calling service credential is verified as a proxy user coming from a
> trusted host verifying Hadoop proxy user ACL on the server side. If proxy
> user ACL allows proxy user to become doAs user. HttpRequest object will
> report REMOTE_USER as doAs user. This feature enables web application logic
> to be written with minimal changes to call Hadoop API with
> UserGroupInformation.doAs() wrapper.
> h2. HTTP Request
> A few possible options:
> 1. Using query parameter to pass doAs user:
> {code:java}
> POST /service?doAs=foobar
> Authorization: [proxy user Kerberos token]
> {code}
> 2. Use HTTP Header to pass doAs user:
> {code:java}
> POST /service
> Authorization: [proxy user Kerberos token]
> x-hadoop-doas: foobar
> {code}
> h2. HTTP Response
> 403 - Forbidden (Including impersonation is not allowed)
> h2. Proxy User ACL requirement
> Proxy user kerberos token maps to a service principal, such as
> yarn/host1.example.com. The host part of the credential and HTTP request
> origin are both validated with *hadoop.proxyuser.yarn.hosts* ACL. doAs user
> group membership or identity is checked with either
> *hadoop.proxyuser.yarn.groups* or *hadoop.proxyuser.yarn.users*. This governs
> the caller is coming from authorized host and belong to authorized group.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16095) Support impersonation for AuthenticationFilter
[
https://issues.apache.org/jira/browse/HADOOP-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HADOOP-16095:
---
Attachment: HADOOP-16095.004.patch
> Support impersonation for AuthenticationFilter
> --
>
> Key: HADOOP-16095
> URL: https://issues.apache.org/jira/browse/HADOOP-16095
> Project: Hadoop Common
> Issue Type: New Feature
> Components: security
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16095.004.patch
>
>
> External services or YARN service may need to call into WebHDFS or YARN REST
> API on behave of the user using web protocols. It would be good to support
> impersonation mechanism in AuthenticationFilter or similar extensions. The
> general design is similar to UserGroupInformation.doAs in RPC layer.
> The calling service credential is verified as a proxy user coming from a
> trusted host verifying Hadoop proxy user ACL on the server side. If proxy
> user ACL allows proxy user to become doAs user. HttpRequest object will
> report REMOTE_USER as doAs user. This feature enables web application logic
> to be written with minimal changes to call Hadoop API with
> UserGroupInformation.doAs() wrapper.
> h2. HTTP Request
> A few possible options:
> 1. Using query parameter to pass doAs user:
> {code:java}
> POST /service?doAs=foobar
> Authorization: [proxy user Kerberos token]
> {code}
> 2. Use HTTP Header to pass doAs user:
> {code:java}
> POST /service
> Authorization: [proxy user Kerberos token]
> x-hadoop-doas: foobar
> {code}
> h2. HTTP Response
> 403 - Forbidden (Including impersonation is not allowed)
> h2. Proxy User ACL requirement
> Proxy user kerberos token maps to a service principal, such as
> yarn/host1.example.com. The host part of the credential and HTTP request
> origin are both validated with *hadoop.proxyuser.yarn.hosts* ACL. doAs user
> group membership or identity is checked with either
> *hadoop.proxyuser.yarn.groups* or *hadoop.proxyuser.yarn.users*. This governs
> the caller is coming from authorized host and belong to authorized group.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16287) KerberosAuthenticationHandler Trusted Proxy Support for Knox
[ https://issues.apache.org/jira/browse/HADOOP-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840528#comment-16840528 ] Eric Yang commented on HADOOP-16287: +1 on patch 007 to include the new ProxyUserAuthenticationFilter. For this to work globally in a web application such as YARN UI or HDFS UI, we must ensure that the same filter mechanism applies to all endpoints. I opened HADOOP-16314 to track the required changes in application code. > KerberosAuthenticationHandler Trusted Proxy Support for Knox > > > Key: HADOOP-16287 > URL: https://issues.apache.org/jira/browse/HADOOP-16287 > Project: Hadoop Common > Issue Type: New Feature > Components: auth >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: HADOOP-16287-001.patch, HADOOP-16287-002.patch, > HADOOP-16287-004.patch, HADOOP-16287-005.patch, HADOOP-16287-006.patch, > HADOOP-16287-007.patch, HADOOP-16827-003.patch > > > Knox passes doAs with end user while accessing RM, WebHdfs Rest Api. > Currently KerberosAuthenticationHandler sets the remote user to Knox. Need > Trusted Proxy Support by reading doAs query parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16314) Make sure all end point URL is covered by the same AuthenticationFilter
Eric Yang created HADOOP-16314: -- Summary: Make sure all end point URL is covered by the same AuthenticationFilter Key: HADOOP-16314 URL: https://issues.apache.org/jira/browse/HADOOP-16314 Project: Hadoop Common Issue Type: Improvement Reporter: Eric Yang In the enclosed spreadsheet, it shows the list of web applications deployed by Hadoop, and filters applied to each entry point. Hadoop web protocol impersonation has been inconsistent. Most of entry point do not support ?doAs parameter. This creates problem for secure gateway like Knox to proxy Hadoop web interface on behave of the end user. When the receiving end does not check for ?doAs flag, web interface would be accessed using proxy user credential. This can lead to all kind of security holes using path traversal to exploit Hadoop. In HADOOP-16287, ProxyUserAuthenticationFilter is proposed as solution to solve the web impersonation problem. This task is to track changes required in Hadoop code base to apply authentication filter globally for each of the web service port. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16161) NetworkTopology#getWeightUsingNetworkLocation return unexpected result
[ https://issues.apache.org/jira/browse/HADOOP-16161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840523#comment-16840523 ] He Xiaoqiao commented on HADOOP-16161: -- Thanks [~elgoiri] for the review and commit. > NetworkTopology#getWeightUsingNetworkLocation return unexpected result > -- > > Key: HADOOP-16161 > URL: https://issues.apache.org/jira/browse/HADOOP-16161 > Project: Hadoop Common > Issue Type: Bug > Components: net >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-16161.001.patch, HADOOP-16161.002.patch, > HADOOP-16161.003.patch, HADOOP-16161.004.patch, HADOOP-16161.005.patch, > HADOOP-16161.006.patch, HADOOP-16161.007.patch, HADOOP-16161.008.patch, > HADOOP-16161.009.patch > > > Consider the following scenario: > 1. there are 4 slaves and topology like: > Rack: /IDC/RACK1 >hostname1 >hostname2 > Rack: /IDC/RACK2 >hostname3 >hostname4 > 2. Reader from hostname1, and calculate weight between reader and [hostname1, > hostname3, hostname4] by #getWeight, and their corresponding values are > [0,4,4] > 3. Reader from client which is not in the topology, and in the same IDC but > in none rack of the topology, and calculate weight between reader and > [hostname1, hostname3, hostname4] by #getWeightUsingNetworkLocation, and > their corresponding values are [2,2,2] > 4. Other different Reader can get the similar results. > The weight result for case #3 is obviously not the expected value, the truth > is [4,4,4]. this issue may cause reader not really following arrange: local > -> local rack -> remote rack. > After dig the detailed implement, the root cause is > #getWeightUsingNetworkLocation only calculate distance between Racks rather > than hosts. > I think we should add constant 2 to correct the weight of > #getWeightUsingNetworkLocation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename
[
https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840347#comment-16840347
]
Steve Loughran commented on HADOOP-15183:
-
HADOOP-15604 is going to be addressed by this: the tracking of parent entries
created during a bulk commit is exactly the same as that during a bulk rename.
The big difference is that for HADOOP-15604 the context has to go all the way
to the committers. I'm doing some reworking of how {{CommitOperations}} is
invoked: there'll be a {{CommitContext}} you request which will contain any
reference to the metastore state, and which exports the commit/abort methods
for the committers to call. WriteOperationHelper will take this reference
during a new commit method.
With this design the nature of the metastore gets as far as
{{CommitOperations}}, but no further. Even that is a bit further than I would
like, but I can't see any way to avoid that
> S3Guard store becomes inconsistent after partial failure of rename
> --
>
> Key: HADOOP-15183
> URL: https://issues.apache.org/jira/browse/HADOOP-15183
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15183-001.patch, HADOOP-15183-002.patch,
> org.apache.hadoop.fs.s3a.auth.ITestAssumeRole-output.txt
>
>
> If an S3A rename() operation fails partway through, such as when the user
> doesn't have permissions to delete the source files after copying to the
> destination, then the s3guard view of the world ends up inconsistent. In
> particular the sequence
> (assuming src/file* is a list of files file1...file10 and read only to
> caller)
>
> # create file rename src/file1 dest/ ; expect AccessDeniedException in the
> delete, dest/file1 will exist
> # delete file dest/file1
> # rename src/file* dest/ ; expect failure
> # list dest; you will not see dest/file1
> You will not see file1 in the listing, presumably because it will have a
> tombstone marker and the update at the end of the rename() didn't take place:
> the old data is still there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jiwq opened a new pull request #821: MAPREDUCE-7205. Changed to the interpret container scheduler kill exit code as a task attempt killing event
jiwq opened a new pull request #821: MAPREDUCE-7205. Changed to the interpret container scheduler kill exit code as a task attempt killing event URL: https://github.com/apache/hadoop/pull/821 [MAPREDUCE-7205](https://issues.apache.org/jira/browse/) MR needs to recognize the special exit code value of -108 and interpret it as a container being killed instead of a container failure. -108: Container was terminated by the ContainerScheduler to make room for another container... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16263) Update BUILDING.txt with macOS native build instructions
[ https://issues.apache.org/jira/browse/HADOOP-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840268#comment-16840268 ] Adam Antal commented on HADOOP-16263: - Sorry didn't have the chance to try this yet, but I hope I can test it this week. > Update BUILDING.txt with macOS native build instructions > > > Key: HADOOP-16263 > URL: https://issues.apache.org/jira/browse/HADOOP-16263 > Project: Hadoop Common > Issue Type: Task >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Minor > Attachments: HADOOP-16263.001.patch, HADOOP-16263.002.patch > > > I recently tried to compile Hadoop native on a Mac and found a few catches, > involving fixing some YARN native compiling issues (YARN-8622, YARN-9487). > Also, need to specify OpenSSL (brewed) header include dir when building > native with maven on a Mac. Should update BUILDING.txt for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16313) multipart/huge file upload tests to look at checksums returned
Steve Loughran created HADOOP-16313:
---
Summary: multipart/huge file upload tests to look at checksums
returned
Key: HADOOP-16313
URL: https://issues.apache.org/jira/browse/HADOOP-16313
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.2.0
Reporter: Steve Loughran
Apparently some third party S3-API object stores have empty etags for multipart
uploads.
We don't have any tests to explore this (just small file uploads).
Proposed
*set {{fs.s3a.etag.checksum.enabled"}} before huge file, MPU and committer tests
* warn (fail?) if nothing comes back.
If true, it means that the store isn't going to be detect changes to a large
file during copy/read
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] supratimdeka commented on issue #812: HDDS-1511. Space tracking for Open Containers in HDDS Volumes. Contributed by Supratim Deka
supratimdeka commented on issue #812: HDDS-1511. Space tracking for Open Containers in HDDS Volumes. Contributed by Supratim Deka URL: https://github.com/apache/hadoop/pull/812#issuecomment-492577036 TestBlockOutputStreamWithFailures appears to be unpredictable - failures are intermittent. TestHddsDatanodeService appears to be broken. both are failing on trunk without the patch. Filed a jira for this issue. remaining tests are passing locally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on issue #820: HDDS-1531. Disable the sync flag by default during chunk writes in Datanode.
mukul1987 commented on issue #820: HDDS-1531. Disable the sync flag by default during chunk writes in Datanode. URL: https://github.com/apache/hadoop/pull/820#issuecomment-492572153 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #820: HDDS-1531. Disable the sync flag by default during chunk writes in Datanode.
hadoop-yetus commented on issue #820: HDDS-1531. Disable the sync flag by default during chunk writes in Datanode. URL: https://github.com/apache/hadoop/pull/820#issuecomment-492537408 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 410 | trunk passed | | +1 | compile | 197 | trunk passed | | +1 | checkstyle | 51 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 901 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 120 | trunk passed | | 0 | spotbugs | 240 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 424 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 398 | the patch passed | | +1 | compile | 201 | the patch passed | | +1 | javac | 201 | the patch passed | | +1 | checkstyle | 56 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 715 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 121 | the patch passed | | +1 | findbugs | 443 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 149 | hadoop-hdds in the patch failed. | | -1 | unit | 1219 | hadoop-ozone in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 5621 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-820/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/820 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 4473a5d7ee5e 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 2d8282b | | Default Java | 1.8.0_191 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-820/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-820/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-820/1/testReport/ | | Max. process+thread count | 4591 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common U: hadoop-hdds/common | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-820/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16287) KerberosAuthenticationHandler Trusted Proxy Support for Knox
[ https://issues.apache.org/jira/browse/HADOOP-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840110#comment-16840110 ] Prabhu Joseph commented on HADOOP-16287: Fixed checkstyle issues in [^HADOOP-16287-007.patch]. > KerberosAuthenticationHandler Trusted Proxy Support for Knox > > > Key: HADOOP-16287 > URL: https://issues.apache.org/jira/browse/HADOOP-16287 > Project: Hadoop Common > Issue Type: New Feature > Components: auth >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: HADOOP-16287-001.patch, HADOOP-16287-002.patch, > HADOOP-16287-004.patch, HADOOP-16287-005.patch, HADOOP-16287-006.patch, > HADOOP-16287-007.patch, HADOOP-16827-003.patch > > > Knox passes doAs with end user while accessing RM, WebHdfs Rest Api. > Currently KerberosAuthenticationHandler sets the remote user to Knox. Need > Trusted Proxy Support by reading doAs query parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16287) KerberosAuthenticationHandler Trusted Proxy Support for Knox
[
https://issues.apache.org/jira/browse/HADOOP-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840105#comment-16840105
]
Hadoop QA commented on HADOOP-16287:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HADOOP-16287 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12968750/HADOOP-16287-007.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 7037ec65c246 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar
18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2d8282b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16251/testReport/ |
| Max. process+thread count | 1717 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16251/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically
