[jira] [Commented] (HDDS-372) There are three buffer copies in BlockOutputStream
[
https://issues.apache.org/jira/browse/HDDS-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809535#comment-16809535
]
Tsz Wo Nicholas Sze commented on HDDS-372:
--
BlockOutputStreamEntry is still using safeBufferByteStringCopy and it is built
by isUnsafeByteOperationsEnabled, i.e. unsafe becomes safe. To avoid this kind
of bug, let's avoid passing the boolean around. We may initialize
ByteStringHelper as below.
{code}
public class ByteStringHelper {
private static final AtomicBoolean initialized = new AtomicBoolean();
private static volatile boolean isUnsafeByteOperationsEnabled;
public static void init(boolean isUnsafeByteOperationsEnabled) {
final boolean set = initialized.compareAndSet(false, true);
if (set) {
ByteStringHelper.isUnsafeByteOperationsEnabled =
isUnsafeByteOperationsEnabled;
} else {
// already initialized, check values
Preconditions.checkState(ByteStringHelper.isUnsafeByteOperationsEnabled
== isUnsafeByteOperationsEnabled);
}
}
public static ByteString getByteString(ByteBuffer buffer) {
return isUnsafeByteOperationsEnabled ?
UnsafeByteOperations.unsafeWrap(buffer) : ByteString.copyFrom(buffer);
}
public static ByteString getByteString(byte[] bytes) {
return isUnsafeByteOperationsEnabled ?
UnsafeByteOperations.unsafeWrap(bytes) : ByteString.copyFrom(bytes);
}
}
{code}
> There are three buffer copies in BlockOutputStream
> --
>
> Key: HDDS-372
> URL: https://issues.apache.org/jira/browse/HDDS-372
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-372.001.patch, HDDS-372.002.patch,
> HDDS-372.003.patch, HDDS-372.20180829.patch
>
>
> Currently, there are three buffer copies in ChunkOutputStream
> # from byte[] to ByteBuffer, and
> # from ByteBuffer to ByteString.
> # from ByteString to ByteBuffer for checskum computation
> We should eliminate the ByteBuffer in the middle.
> For zero copy io, we should support WritableByteChannel instead of
> OutputStream. It won't be done in this JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14313) Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du
[
https://issues.apache.org/jira/browse/HDFS-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809530#comment-16809530
]
Hadoop QA commented on HDFS-14313:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 56s{color} | {color:orange} root: The patch generated 12 new + 245 unchanged
- 1 fixed = 257 total (was 246) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 59s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
35s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
41s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14313 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964800/HDFS-14313.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6a24c792fe5b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7b5b783 |
| maven | version:
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222819=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222819 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 04/Apr/19 05:50 Start Date: 04/Apr/19 05:50 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222819) Time Spent: 5h (was: 4h 50m) > Implement Ratis Snapshots on OM > --- > > Key: HDDS-1339 > URL: https://issues.apache.org/jira/browse/HDDS-1339 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > For bootstrapping and restarting OMs, we need to implement snapshots in OM. > The OM state maintained by RocksDB will be checkpoint-ed on demand. Ratis > snapshots will only preserve the last applied log index by the State Machine > on disk. This index will be stored in file in the OM metadata dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222818=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222818 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 04/Apr/19 05:49 Start Date: 04/Apr/19 05:49 Worklog Time Spent: 10m Work Description: hanishakoneru commented on issue #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651#issuecomment-479759384 Thank you @bharatviswa504 for the reviews. The CI unit and acceptance test failure is not related to this PR. I will merge the PR with trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222818) Time Spent: 4h 50m (was: 4h 40m) > Implement Ratis Snapshots on OM > --- > > Key: HDDS-1339 > URL: https://issues.apache.org/jira/browse/HDDS-1339 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > For bootstrapping and restarting OMs, we need to implement snapshots in OM. > The OM state maintained by RocksDB will be checkpoint-ed on demand. Ratis > snapshots will only preserve the last applied log index by the State Machine > on disk. This index will be stored in file in the OM metadata dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[ https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809519#comment-16809519 ] Anoop Sam John commented on HDFS-14401: --- Now the new config "dfs.datanode.cache.loader.class" is mandatory to use the pmem based cache. I know this was added because we have the Java based impl and another subtack is planning for a native impl (based on availability of PMDK lib) Being a user I think it is an unwanted overhead. When we have a native impl and pure Java impl can HDFS select which loader to be used automatically? If the native lib is available in a node, the native loader only can be used? The native impl is added because it is much better performing for reads and writes from/to cache. So when the node is ready for native loader and that is a better performing one, I am not sure why being a user I should try to use the less performing loader version. > Refine the implementation for HDFS cache on SCM > --- > > Key: HDFS-14401 > URL: https://issues.apache.org/jira/browse/HDFS-14401 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > > In this Jira, we will refine the implementation for HDFS cache on SCM, such > as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume > selection impl; 3) Clean up MapppableBlockLoader interface; etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1189) Recon Aggregate DB schema and ORM
[ https://issues.apache.org/jira/browse/HDDS-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809505#comment-16809505 ] Siddharth Wagle edited comment on HDDS-1189 at 4/4/19 4:53 AM: --- [~arpitagarwal]/[~linyiqun] Rebased patch attached. was (Author: swagle): [~arpitagarwal] Rebased patch attached. > Recon Aggregate DB schema and ORM > - > > Key: HDDS-1189 > URL: https://issues.apache.org/jira/browse/HDDS-1189 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1189.01.patch, HDDS-1189.02.patch, > HDDS-1189.03.patch, HDDS-1189.04.patch, HDDS-1189.05.patch, > HDDS-1189.06.patch, HDDS-1189.08.patch > > > _Objectives_ > - Define V1 of the db schema for recon service > - The current proposal is to use jOOQ as the ORM for SQL interaction. For two > main reasons: a) powerful DSL for querying, that abstracts out SQL dialects, > b) Allows code to schema and schema to code seamless transition, critical for > creating DDL through the code and unit testing across versions of the > application. > - Add e2e unit tests suite for Recon entities, created based on the design doc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1189) Recon Aggregate DB schema and ORM
[ https://issues.apache.org/jira/browse/HDDS-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809505#comment-16809505 ] Siddharth Wagle commented on HDDS-1189: --- [~arpitagarwal] Rebased patch attached. > Recon Aggregate DB schema and ORM > - > > Key: HDDS-1189 > URL: https://issues.apache.org/jira/browse/HDDS-1189 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1189.01.patch, HDDS-1189.02.patch, > HDDS-1189.03.patch, HDDS-1189.04.patch, HDDS-1189.05.patch, > HDDS-1189.06.patch, HDDS-1189.08.patch > > > _Objectives_ > - Define V1 of the db schema for recon service > - The current proposal is to use jOOQ as the ORM for SQL interaction. For two > main reasons: a) powerful DSL for querying, that abstracts out SQL dialects, > b) Allows code to schema and schema to code seamless transition, critical for > creating DDL through the code and unit testing across versions of the > application. > - Add e2e unit tests suite for Recon entities, created based on the design doc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14327) Using FQDN instead of IP to access servers with DNS resolving
[ https://issues.apache.org/jira/browse/HDFS-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809503#comment-16809503 ] Íñigo Goiri commented on HDFS-14327: Nope, it always complains... I don't remember last time it succeeded. This is done. > Using FQDN instead of IP to access servers with DNS resolving > - > > Key: HDFS-14327 > URL: https://issues.apache.org/jira/browse/HDFS-14327 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14327.001.patch, HDFS-14327.002.patch > > > With [HDFS-14118|https://issues.apache.org/jira/browse/HDFS-14118], clients > can get the IP of the servers (NN/Routers) and use the IP addresses to access > the machine. This will fail in secure environment as Kerberos is using the > domain name (FQDN) in the principal so it won't recognize the IP addresses. > This task is mainly adding a reverse look up on the current basis and get the > domain name after the IP is fetched. After that clients will still use the > domain name to access the servers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1189) Recon Aggregate DB schema and ORM
[ https://issues.apache.org/jira/browse/HDDS-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated HDDS-1189: -- Attachment: HDDS-1189.08.patch > Recon Aggregate DB schema and ORM > - > > Key: HDDS-1189 > URL: https://issues.apache.org/jira/browse/HDDS-1189 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1189.01.patch, HDDS-1189.02.patch, > HDDS-1189.03.patch, HDDS-1189.04.patch, HDDS-1189.05.patch, > HDDS-1189.06.patch, HDDS-1189.08.patch > > > _Objectives_ > - Define V1 of the db schema for recon service > - The current proposal is to use jOOQ as the ORM for SQL interaction. For two > main reasons: a) powerful DSL for querying, that abstracts out SQL dialects, > b) Allows code to schema and schema to code seamless transition, critical for > creating DDL through the code and unit testing across versions of the > application. > - Add e2e unit tests suite for Recon entities, created based on the design doc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809501#comment-16809501
]
Íñigo Goiri commented on HDFS-13853:
Thanks [~ayushtkn] for the clarification, one last comment: In line 649, let's
avoid the concat; is there a way to avoid a double colon? Maybe:
{code}
msg = "Unable to parse arguments: no value provided for "
+ parameters[i - 1];
{code}
I would leave the {{-}} there and no need for substring.
> RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
> --
>
> Key: HDFS-13853
> URL: https://issues.apache.org/jira/browse/HDFS-13853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-13853-HDFS-13891-01.patch,
> HDFS-13853-HDFS-13891-02.patch, HDFS-13853-HDFS-13891-03.patch,
> HDFS-13853-HDFS-13891-04.patch, HDFS-13853-HDFS-13891-05.patch,
> HDFS-13853-HDFS-13891-06.patch, HDFS-13853-HDFS-13891-07.patch,
> HDFS-13853-HDFS-13891-08.patch
>
>
> {code:java}
> // Create a new entry
> Map destMap = new LinkedHashMap<>();
> for (String ns : nss) {
> destMap.put(ns, dest);
> }
> MountTable newEntry = MountTable.newInstance(mount, destMap);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1349) Remove watchClient from XceiverClientRatis
[ https://issues.apache.org/jira/browse/HDDS-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809500#comment-16809500 ] Lokesh Jain commented on HDDS-1349: --- [~shashikant] Thanks for working on this! The patch looks good to me. +1. > Remove watchClient from XceiverClientRatis > -- > > Key: HDDS-1349 > URL: https://issues.apache.org/jira/browse/HDDS-1349 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.5.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1349.000.patch > > > WatchForCommit now bypasses the sliding window of RaftClient. and hence > creating a new raft client for calling watchForCommit is not required as it > won't block any subsequent calls. This Jira aims to remove the watchClient > from XceiverClientRatis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1333) OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security classes
[
https://issues.apache.org/jira/browse/HDDS-1333?focusedWorklogId=222806=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222806
]
ASF GitHub Bot logged work on HDDS-1333:
Author: ASF GitHub Bot
Created on: 04/Apr/19 04:39
Start Date: 04/Apr/19 04:39
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #653: HDDS-1333.
OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security
classes
URL: https://github.com/apache/hadoop/pull/653#discussion_r272015093
##
File path: hadoop-ozone/dist/src/main/compose/ozonefs/docker-compose.yaml
##
@@ -49,21 +49,53 @@ services:
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
command: ["/opt/hadoop/bin/ozone","scm"]
- hadoop3:
+ hadoop32:
image: flokkr/hadoop:3.1.0
Review comment:
Agree, let's fix that post 0.4.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222806)
Time Spent: 6h 10m (was: 6h)
> OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security
> classes
> -
>
> Key: HDDS-1333
> URL: https://issues.apache.org/jira/browse/HDDS-1333
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> The current ozonefs compatibility layer is broken by: HDDS-1299.
> The spark jobs (including hadoop 2.7) can't be executed any more:
> {code}
> 2019-03-25 09:50:08 INFO StateStoreCoordinatorRef:54 - Registered
> StateStoreCoordinator endpoint
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/crypto/key/KeyProviderTokenIssuer
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at
> org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:45)
> at
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:332)
> at
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
> at
> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
> at
> org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:715)
> at
> org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:757)
> at
> org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:724)
> at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:45)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Commented] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809490#comment-16809490
]
Hadoop QA commented on HDFS-13853:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
27s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
53s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}126m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 30s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}225m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
|
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13853 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964791/HDFS-13853-HDFS-13891-07.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c1c997b05290 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[ https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=222802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222802 ] ASF GitHub Bot logged work on HDDS-976: --- Author: ASF GitHub Bot Created on: 04/Apr/19 04:14 Start Date: 04/Apr/19 04:14 Worklog Time Spent: 10m Work Description: xiaoyuyao commented on issue #661: HDDS-976: Parse network topology from yaml file URL: https://github.com/apache/hadoop/pull/661#issuecomment-479743474 Thanks @cjjnjust for working on this. Patch LGTM overall, just few minor issues commented inline. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222802) Time Spent: 1.5h (was: 1h 20m) > Support YAML format network topology cluster definition > --- > > Key: HDDS-976 > URL: https://issues.apache.org/jira/browse/HDDS-976 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Sammi Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Attachments: NetworkTopologyDefault.yaml > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222797 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 04/Apr/19 04:07 Start Date: 04/Apr/19 04:07 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651#issuecomment-479742555 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 24 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 59 | Maven dependency ordering for branch | | +1 | mvninstall | 1006 | trunk passed | | +1 | compile | 964 | trunk passed | | +1 | checkstyle | 192 | trunk passed | | -1 | mvnsite | 37 | ozone-manager in trunk failed. | | +1 | shadedclient | 1092 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | -1 | findbugs | 29 | ozone-manager in trunk failed. | | +1 | javadoc | 122 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 129 | the patch passed | | +1 | compile | 938 | the patch passed | | +1 | javac | 938 | the patch passed | | +1 | checkstyle | 209 | the patch passed | | +1 | mvnsite | 148 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 614 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 198 | the patch passed | | +1 | javadoc | 118 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 82 | common in the patch passed. | | +1 | unit | 39 | common in the patch passed. | | -1 | unit | 1545 | integration-test in the patch failed. | | +1 | unit | 50 | ozone-manager in the patch passed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 7827 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-651/7/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/651 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 78ec5a2dac0e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7b5b783 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/7/artifact/out/branch-mvnsite-hadoop-ozone_ozone-manager.txt | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/7/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/7/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/7/testReport/ | | Max. process+thread count | 3676 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/7/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222797) Time Spent: 4h 40m (was: 4.5h) > Implement Ratis Snapshots on OM > --- > > Key: HDDS-1339 > URL: https://issues.apache.org/jira/browse/HDDS-1339 > Project: Hadoop Distributed Data Store > Issue Type:
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[
https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=222792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222792
]
ASF GitHub Bot logged work on HDDS-976:
---
Author: ASF GitHub Bot
Created on: 04/Apr/19 04:01
Start Date: 04/Apr/19 04:01
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #661: HDDS-976:
Parse network topology from yaml file
URL: https://github.com/apache/hadoop/pull/661#discussion_r272010158
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaManager.java
##
@@ -59,13 +59,20 @@ public void init(Configuration conf) {
/**
* Load schemas from network topology schema configuration file
*/
+String schemaFileType = conf.get(
+ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE_TYPE);
+
String schemaFile = conf.get(
ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE,
ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE_DEFAULT);
NodeSchemaLoadResult result;
try {
- result = NodeSchemaLoader.getInstance().loadSchemaFromFile(schemaFile);
+ if (schemaFileType.compareTo("yaml") == 0) {
+result = NodeSchemaLoader.getInstance().loadSchemaFromYaml(schemaFile);
+ } else {
+result = NodeSchemaLoader.getInstance().loadSchemaFromFile(schemaFile);
Review comment:
Maybe change this to loadSchemaFromXml?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222792)
Time Spent: 1h 20m (was: 1h 10m)
> Support YAML format network topology cluster definition
> ---
>
> Key: HDDS-976
> URL: https://issues.apache.org/jira/browse/HDDS-976
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Attachments: NetworkTopologyDefault.yaml
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[
https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=222791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222791
]
ASF GitHub Bot logged work on HDDS-976:
---
Author: ASF GitHub Bot
Created on: 04/Apr/19 04:00
Start Date: 04/Apr/19 04:00
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #661: HDDS-976:
Parse network topology from yaml file
URL: https://github.com/apache/hadoop/pull/661#discussion_r272010041
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaManager.java
##
@@ -59,13 +59,20 @@ public void init(Configuration conf) {
/**
* Load schemas from network topology schema configuration file
*/
+String schemaFileType = conf.get(
+ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE_TYPE);
+
String schemaFile = conf.get(
ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE,
ScmConfigKeys.OZONE_SCM_NETWORK_TOPOLOGY_SCHEMA_FILE_DEFAULT);
NodeSchemaLoadResult result;
try {
- result = NodeSchemaLoader.getInstance().loadSchemaFromFile(schemaFile);
+ if (schemaFileType.compareTo("yaml") == 0) {
Review comment:
Can we make the schema type string case insensitive?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222791)
Time Spent: 1h 10m (was: 1h)
> Support YAML format network topology cluster definition
> ---
>
> Key: HDDS-976
> URL: https://issues.apache.org/jira/browse/HDDS-976
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Attachments: NetworkTopologyDefault.yaml
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13248) RBF: Namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809478#comment-16809478
]
He Xiaoqiao commented on HDFS-13248:
Thanks [~elgoiri], [~ayushtkn]
To [~elgoiri],
{quote}we need a full design docs.{quote}
Correct, design docs will attach later. I am not very familiar with solution
`modifying the RPC protocol`, anyone would like to explain will very helpful.
To [~ayushtkn],
{quote}Saw this getAdditionalDatanode() having client name parameter in(need to
dig in more) Wouldn't that work for us?{quote}
Parameter #clientName about {{getAdditionalDatanode}} just a name tag and do
not include any hostname/IP unless we change it.
{code:java}
this.clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" +
ThreadLocalRandom.current().nextInt() + "_" +
Thread.currentThread().getId();
{code}
{quote}Do we need to do with scenarios like HBASE-22103?{quote}
If extend protocol, we need compatible with all current interface.
Please correct if something wrong.
> RBF: Namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Wu
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-13248.000.patch, HDFS-13248.001.patch,
> HDFS-13248.002.patch, HDFS-13248.003.patch, HDFS-13248.004.patch,
> HDFS-13248.005.patch, clientMachine-call-path.jpeg, debug-info-1.jpeg,
> debug-info-2.jpeg
>
>
> When execute a put operation via router, the NameNode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both NameNode and Router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[
https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=222787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222787
]
ASF GitHub Bot logged work on HDDS-976:
---
Author: ASF GitHub Bot
Created on: 04/Apr/19 03:47
Start Date: 04/Apr/19 03:47
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #661: HDDS-976:
Parse network topology from yaml file
URL: https://github.com/apache/hadoop/pull/661#discussion_r272008319
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaLoader.java
##
@@ -165,6 +169,81 @@ private NodeSchemaLoadResult loadSchema(File schemaFile)
throws
return schemaList;
}
+ /**
+ * Load user defined network layer schemas from a YAML configuration file.
+ * @param schemaFilePath path of schema file
+ * @return all valid node schemas defined in schema file
+ */
+ public NodeSchemaLoadResult loadSchemaFromYaml(String schemaFilePath)
+ throws IllegalArgumentException {
+try {
+ File schemaFile = new File(schemaFilePath);
+ if (!schemaFile.exists()) {
+String msg = "Network topology layer schema file " + schemaFilePath +
+" is not found.";
+LOG.warn(msg);
+throw new IllegalArgumentException(msg);
+ }
+ return loadSchemaFromYaml(schemaFile);
+} catch (Exception e) {
+ throw new IllegalArgumentException("Fail to load network topology node"
+ + " schema file: " + schemaFilePath + " , error:" +
e.getMessage());
+}
+ }
+
+ /**
+ * Load network topology layer schemas from a YAML configuration file.
+ * @param schemaFile schema file
+ * @return all valid node schemas defined in schema file
+ * @throws ParserConfigurationException ParserConfigurationException happen
+ * @throws IOException no such schema file
+ * @throws SAXException xml file has some invalid elements
+ * @throws IllegalArgumentException xml file content is logically invalid
+ */
+ private NodeSchemaLoadResult loadSchemaFromYaml(File schemaFile) {
+LOG.info("Loading network topology layer schema file " + schemaFile);
+NodeSchemaLoadResult finalSchema;
+
+try {
+ FileInputStream fileInputStream = new FileInputStream(schemaFile);
Review comment:
Can we use try-with-resource to ensure the FileInputStream is closed
properly even Exception is thrown on line 210?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222787)
Time Spent: 1h (was: 50m)
> Support YAML format network topology cluster definition
> ---
>
> Key: HDDS-976
> URL: https://issues.apache.org/jira/browse/HDDS-976
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Attachments: NetworkTopologyDefault.yaml
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[
https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=222786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222786
]
ASF GitHub Bot logged work on HDDS-976:
---
Author: ASF GitHub Bot
Created on: 04/Apr/19 03:46
Start Date: 04/Apr/19 03:46
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #661: HDDS-976:
Parse network topology from yaml file
URL: https://github.com/apache/hadoop/pull/661#discussion_r272008319
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaLoader.java
##
@@ -165,6 +169,81 @@ private NodeSchemaLoadResult loadSchema(File schemaFile)
throws
return schemaList;
}
+ /**
+ * Load user defined network layer schemas from a YAML configuration file.
+ * @param schemaFilePath path of schema file
+ * @return all valid node schemas defined in schema file
+ */
+ public NodeSchemaLoadResult loadSchemaFromYaml(String schemaFilePath)
+ throws IllegalArgumentException {
+try {
+ File schemaFile = new File(schemaFilePath);
+ if (!schemaFile.exists()) {
+String msg = "Network topology layer schema file " + schemaFilePath +
+" is not found.";
+LOG.warn(msg);
+throw new IllegalArgumentException(msg);
+ }
+ return loadSchemaFromYaml(schemaFile);
+} catch (Exception e) {
+ throw new IllegalArgumentException("Fail to load network topology node"
+ + " schema file: " + schemaFilePath + " , error:" +
e.getMessage());
+}
+ }
+
+ /**
+ * Load network topology layer schemas from a YAML configuration file.
+ * @param schemaFile schema file
+ * @return all valid node schemas defined in schema file
+ * @throws ParserConfigurationException ParserConfigurationException happen
+ * @throws IOException no such schema file
+ * @throws SAXException xml file has some invalid elements
+ * @throws IllegalArgumentException xml file content is logically invalid
+ */
+ private NodeSchemaLoadResult loadSchemaFromYaml(File schemaFile) {
+LOG.info("Loading network topology layer schema file " + schemaFile);
+NodeSchemaLoadResult finalSchema;
+
+try {
+ FileInputStream fileInputStream = new FileInputStream(schemaFile);
Review comment:
Can we use try-with-resource to ensure the FileInputStream is closed
properly?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222786)
Time Spent: 50m (was: 40m)
> Support YAML format network topology cluster definition
> ---
>
> Key: HDDS-976
> URL: https://issues.apache.org/jira/browse/HDDS-976
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Attachments: NetworkTopologyDefault.yaml
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-976) Support YAML format network topology cluster definition
[
https://issues.apache.org/jira/browse/HDDS-976?focusedWorklogId=222785=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222785
]
ASF GitHub Bot logged work on HDDS-976:
---
Author: ASF GitHub Bot
Created on: 04/Apr/19 03:46
Start Date: 04/Apr/19 03:46
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #661: HDDS-976:
Parse network topology from yaml file
URL: https://github.com/apache/hadoop/pull/661#discussion_r272008223
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/net/NodeSchemaLoader.java
##
@@ -165,6 +169,81 @@ private NodeSchemaLoadResult loadSchema(File schemaFile)
throws
return schemaList;
}
+ /**
+ * Load user defined network layer schemas from a YAML configuration file.
+ * @param schemaFilePath path of schema file
+ * @return all valid node schemas defined in schema file
+ */
+ public NodeSchemaLoadResult loadSchemaFromYaml(String schemaFilePath)
+ throws IllegalArgumentException {
+try {
+ File schemaFile = new File(schemaFilePath);
+ if (!schemaFile.exists()) {
+String msg = "Network topology layer schema file " + schemaFilePath +
+" is not found.";
+LOG.warn(msg);
+throw new IllegalArgumentException(msg);
+ }
+ return loadSchemaFromYaml(schemaFile);
+} catch (Exception e) {
+ throw new IllegalArgumentException("Fail to load network topology node"
+ + " schema file: " + schemaFilePath + " , error:" +
e.getMessage());
+}
+ }
+
+ /**
+ * Load network topology layer schemas from a YAML configuration file.
+ * @param schemaFile schema file
+ * @return all valid node schemas defined in schema file
+ * @throws ParserConfigurationException ParserConfigurationException happen
+ * @throws IOException no such schema file
+ * @throws SAXException xml file has some invalid elements
+ * @throws IllegalArgumentException xml file content is logically invalid
+ */
+ private NodeSchemaLoadResult loadSchemaFromYaml(File schemaFile) {
+LOG.info("Loading network topology layer schema file " + schemaFile);
Review comment:
NIT: can we use parameterized log4j like below
LOG.info("Loading network topology layer schema file {}", schemaFile);
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222785)
Time Spent: 40m (was: 0.5h)
> Support YAML format network topology cluster definition
> ---
>
> Key: HDDS-976
> URL: https://issues.apache.org/jira/browse/HDDS-976
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Sammi Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Attachments: NetworkTopologyDefault.yaml
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809473#comment-16809473
]
Hadoop QA commented on HDFS-13853:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
25s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
54s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
41s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}166m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13853 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964793/HDFS-13853-HDFS-13891-08.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c1b663aba4b0 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / dea3798 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-12345) Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
[
https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809448#comment-16809448
]
Hadoop QA commented on HDFS-12345:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 13m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-tools hadoop-dist . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 32s{color}
| {color:red} root generated 2 new + 1496 unchanged - 0 fixed = 1498 total (was
1496) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 6s{color} | {color:orange} root: The patch generated 80 new + 0 unchanged -
0 fixed = 80 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 13m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
1s{color} | {color:red} The patch generated 50 new + 1 unchanged - 0 fixed = 51
total (was 1) {color} |
| {color:orange}-0{color} | {color:orange} shelldocs {color} | {color:orange}
0m 17s{color} | {color:orange} The patch generated 2 new + 236 unchanged - 0
fixed = 238 total (was 236) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
21s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-assemblies hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-dist
hadoop-tools/hadoop-dynamometer hadoop-tools hadoop-dist . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}141m 12s{color}
| {color:red} root in the patch
[jira] [Updated] (HDFS-14313) Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du
[ https://issues.apache.org/jira/browse/HDFS-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14313: --- Attachment: HDFS-14313.001.patch > Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory > instead of df/du > > > Key: HDFS-14313 > URL: https://issues.apache.org/jira/browse/HDFS-14313 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, performance >Affects Versions: 2.6.0, 2.7.0, 2.8.0, 2.9.0, 3.0.0, 3.1.0 >Reporter: Lisheng Sun >Priority: Major > Attachments: HDFS-14313.000.patch, HDFS-14313.001.patch > > > There are two ways of DU/DF getting used space that are insufficient. > # Running DU across lots of disks is very expensive and running all of the > processes at the same time creates a noticeable IO spike. > # Running DF is inaccurate when the disk sharing by multiple datanode or > other servers. > Getting hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfos in memory > is very small and accurate. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809437#comment-16809437
]
Akira Ajisaka commented on HDFS-14369:
--
Run precommit job again
https://builds.apache.org/job/PreCommit-HDFS-Build/26580/
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14369) RBF: Fix trailing "/" for webhdfs
[
https://issues.apache.org/jira/browse/HDFS-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809434#comment-16809434
]
Akira Ajisaka commented on HDFS-14369:
--
Rebased HDFS-13891. HDFS-13891 had
https://github.com/apache/hadoop/commit/bc6fe7ad45986410afa1581572272913aa93e5ec
but the commit does not exist in trunk because of the conflict between gitbox
repository and github repository (INFRA-17947). I skipped the commit and
rebased successfully.
> RBF: Fix trailing "/" for webhdfs
> -
>
> Key: HDFS-14369
> URL: https://issues.apache.org/jira/browse/HDFS-14369
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-14369-HDFS-13891-regressiontest-001.patch,
> HDFS-14369-HDFS-13891.001.patch, HDFS-14369-HDFS-13891.002.patch,
> HDFS-14369-HDFS-13891.003.patch, HDFS-14369-HDFS-13891.004.patch
>
>
> WebHDFS doesn't trim trailing slash causing discrepancy in operations.
> Example below
> --
> Using HDFS API, two directory are listed.
> {code}
> $ hdfs dfs -ls hdfs://:/tmp/
> Found 2 items
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp1
> drwxrwxrwx - hdfs supergroup 0 2018-11-09 17:50
> hdfs://:/tmp/tmp2
> {code}
> Using WebHDFS API, only one directory is listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp/?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16387,"group":"supergroup","length":0,"modificationTime":1552016766769,"owner":"hdfs","pathSuffix":"tmp1","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
> ]}}
> {code}
> The mount table is as follows:
> {code}
> $ hdfs dfsrouteradmin -ls /tmp
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /tmp ns1->/tmp aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp1 ns1->/tmp/tmp1aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> /tmp/tmp2 ns2->/tmp/tmp2aajisaka
> users rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> {code}
> Without trailing thrash, two directories are listed.
> {code}
> $ curl -u : --negotiate -i
> "http://:50071/webhdfs/v1/tmp?op=LISTSTATUS"
> (snip)
> {"FileStatuses":{"FileStatus":[
> {"accessTime":1541753421917,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753421917,"owner":"hdfs","pathSuffix":"tmp1","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"},
> {"accessTime":1541753429812,"blockSize":0,"childrenNum":0,"fileId":0,"group":"supergroup","length":0,"modificationTime":1541753429812,"owner":"hdfs","pathSuffix":"tmp2","permission":"777","replication":0,"storagePolicy":0,"symlink":"","type":"DIRECTORY"}
> ]}}
> {code}
> [~ajisakaa] Thanks for reporting this, I borrowed the text from
> HDFS-13972
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14407) Fix mistakenly used of SLF4j or Log4j2 logging in DatasetVolumeChecker#checkAllVolumes
[
https://issues.apache.org/jira/browse/HDFS-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809365#comment-16809365
]
Wanqiang Ji edited comment on HDFS-14407 at 4/4/19 1:49 AM:
Thanks [~giovanni.fumarola] for the review.
was (Author: jiwq):
Thank [~giovanni.fumarola] for the review.
> Fix mistakenly used of SLF4j or Log4j2 logging in
> DatasetVolumeChecker#checkAllVolumes
> --
>
> Key: HDFS-14407
> URL: https://issues.apache.org/jira/browse/HDFS-14407
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Minor
> Attachments: HDFS-14407.001.patch
>
>
> Wrong:
> {code:java}
> LOG.warn("checkAllVolumes timed out after {} ms" +
> maxAllowedTimeForCheckMs);
> {code}
> Correct:
> {code:java}
> LOG.warn("checkAllVolumes timed out after {} ms",
> maxAllowedTimeForCheckMs);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809427#comment-16809427
]
Wei-Chiu Chuang commented on HDFS-13596:
I am reviewing the patch.
{quote}This issue occurs because namenode writes new layout audit log during
upgrading ,but standby namenode can not parse new layout audit log. So We can
writes audit log according to the current layout version.
{quote}
I'm pretty sure you meant to say "edit log" instead of "audit log".
I think not being able to accept EC requests prior to the completion of
upgrade, is a reasonable trade-off. You can check layout version within
{{FSNamesystem#startFileInt}}, reject when CreateFlag.SHOULD_REPLICATE is false
or ecPolicyName is not empty. Other EC RPCs that should be checked include
setErasureCodingPolicy.
No tests. But you have done the manual test so that's ok.
Further notes: if we want to support rolling upgrades, we should define the
"to" and "from" version supported. I've not done any rolling upgrade test
myself. [~ferhui] what "to" and "from" versions do you have? I think if we can
support 2.8 it'll make the most of the community happy.
Release note or documentation, please.
Things that should be documented – minimum supported versions, caveats,
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Critical
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222743 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 04/Apr/19 01:20 Start Date: 04/Apr/19 01:20 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651#issuecomment-479713962 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 24 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 64 | Maven dependency ordering for branch | | +1 | mvninstall | 1011 | trunk passed | | +1 | compile | 963 | trunk passed | | +1 | checkstyle | 191 | trunk passed | | +1 | mvnsite | 217 | trunk passed | | +1 | shadedclient | 1135 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 213 | trunk passed | | +1 | javadoc | 169 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 61 | Maven dependency ordering for patch | | -1 | mvninstall | 25 | integration-test in the patch failed. | | +1 | compile | 923 | the patch passed | | +1 | javac | 923 | the patch passed | | +1 | checkstyle | 193 | the patch passed | | +1 | mvnsite | 171 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 2 | The patch has no ill-formed XML file. | | +1 | shadedclient | 656 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 214 | the patch passed | | +1 | javadoc | 139 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 73 | common in the patch passed. | | +1 | unit | 40 | common in the patch passed. | | -1 | unit | 1095 | integration-test in the patch failed. | | +1 | unit | 50 | ozone-manager in the patch passed. | | +1 | asflicense | 43 | The patch does not generate ASF License warnings. | | | | 7661 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestContainerStateMachineFailures | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-651/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/651 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux abb72d223c51 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7b5b783 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/6/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/6/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/6/testReport/ | | Max. process+thread count | 3897 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/6/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222743) Time Spent: 4.5h (was: 4h 20m) > Implement Ratis Snapshots on OM > --- > > Key: HDDS-1339 > URL:
[jira] [Updated] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-13853:
Attachment: HDFS-13853-HDFS-13891-08.patch
> RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
> --
>
> Key: HDFS-13853
> URL: https://issues.apache.org/jira/browse/HDFS-13853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-13853-HDFS-13891-01.patch,
> HDFS-13853-HDFS-13891-02.patch, HDFS-13853-HDFS-13891-03.patch,
> HDFS-13853-HDFS-13891-04.patch, HDFS-13853-HDFS-13891-05.patch,
> HDFS-13853-HDFS-13891-06.patch, HDFS-13853-HDFS-13891-07.patch,
> HDFS-13853-HDFS-13891-08.patch
>
>
> {code:java}
> // Create a new entry
> Map destMap = new LinkedHashMap<>();
> for (String ns : nss) {
> destMap.put(ns, dest);
> }
> MountTable newEntry = MountTable.newInstance(mount, destMap);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14397) Backport HADOOP-15684 to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809404#comment-16809404 ] Chen Liang commented on HDFS-14397: --- +1 on v001 patch, I've committed to branch-2, thanks for the contribution [~csun]! > Backport HADOOP-15684 to branch-2 > - > > Key: HDFS-14397 > URL: https://issues.apache.org/jira/browse/HDFS-14397 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HDFS-14397-branch-2.000.patch, > HDFS-14397-branch-2.001.patch > > > As multi-SBN feature is already backported to branch-2, this is a follow-up > to backport HADOOP-15684. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14397) Backport HADOOP-15684 to branch-2
[ https://issues.apache.org/jira/browse/HDFS-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14397: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Backport HADOOP-15684 to branch-2 > - > > Key: HDFS-14397 > URL: https://issues.apache.org/jira/browse/HDFS-14397 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HDFS-14397-branch-2.000.patch, > HDFS-14397-branch-2.001.patch > > > As multi-SBN feature is already backported to branch-2, this is a follow-up > to backport HADOOP-15684. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809402#comment-16809402
]
Ayush Saxena commented on HDFS-13853:
-
{quote}There is also {{Boolean#parseBoolean()}}; any advantage using that?
{quote}
I used that only in the previous versions, but as you said we should have
proper exception in the case of non true false. This method didn't do so, It
just checks for true, if the value is true it returns true else it returns
false.
{quote}I'm guessing the exception for {{DestinationOrder#valueOf()}} is a
runtime one? Otherwise, we could just let the exception pass.
{quote}
That seems runtime only. But the exception message isn't that good.
update: No enum constant
org.apache.hadoop.hdfs.server.federation.resolver.order.DestinationOrder.Invalid
Anyhow to polish the exception we need to handle it somewhere, so I guess this
should be OK
> RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
> --
>
> Key: HDFS-13853
> URL: https://issues.apache.org/jira/browse/HDFS-13853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-13853-HDFS-13891-01.patch,
> HDFS-13853-HDFS-13891-02.patch, HDFS-13853-HDFS-13891-03.patch,
> HDFS-13853-HDFS-13891-04.patch, HDFS-13853-HDFS-13891-05.patch,
> HDFS-13853-HDFS-13891-06.patch, HDFS-13853-HDFS-13891-07.patch
>
>
> {code:java}
> // Create a new entry
> Map destMap = new LinkedHashMap<>();
> for (String ns : nss) {
> destMap.put(ns, dest);
> }
> MountTable newEntry = MountTable.newInstance(mount, destMap);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14327) Using FQDN instead of IP to access servers with DNS resolving
[ https://issues.apache.org/jira/browse/HDFS-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809398#comment-16809398 ] Fengnan Li commented on HDFS-14327: --- Thanks for the commit [~elgoiri] for the hudson error: [ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.3.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' -> [Help 1] [ERROR] Do I need to do something about it? > Using FQDN instead of IP to access servers with DNS resolving > - > > Key: HDFS-14327 > URL: https://issues.apache.org/jira/browse/HDFS-14327 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14327.001.patch, HDFS-14327.002.patch > > > With [HDFS-14118|https://issues.apache.org/jira/browse/HDFS-14118], clients > can get the IP of the servers (NN/Routers) and use the IP addresses to access > the machine. This will fail in secure environment as Kerberos is using the > domain name (FQDN) in the principal so it won't recognize the IP addresses. > This task is mainly adding a reverse look up on the current basis and get the > domain name after the IP is fetched. After that clients will still use the > domain name to access the servers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-13853:
Attachment: HDFS-13853-HDFS-13891-07.patch
> RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
> --
>
> Key: HDFS-13853
> URL: https://issues.apache.org/jira/browse/HDFS-13853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-13853-HDFS-13891-01.patch,
> HDFS-13853-HDFS-13891-02.patch, HDFS-13853-HDFS-13891-03.patch,
> HDFS-13853-HDFS-13891-04.patch, HDFS-13853-HDFS-13891-05.patch,
> HDFS-13853-HDFS-13891-06.patch, HDFS-13853-HDFS-13891-07.patch
>
>
> {code:java}
> // Create a new entry
> Map destMap = new LinkedHashMap<>();
> for (String ns : nss) {
> destMap.put(ns, dest);
> }
> MountTable newEntry = MountTable.newInstance(mount, destMap);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222715
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 04/Apr/19 00:00
Start Date: 04/Apr/19 00:00
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271976741
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1594,6 +1598,107 @@ public void createVolume(OmVolumeArgs args) throws
IOException {
}
}
+ @Override
+ public VolumeList startCreateVolume(OmVolumeArgs args) throws IOException {
+try {
+ // TODO: Need to add metrics and Audit log for HA requests
+ if(isAclEnabled) {
Review comment:
Will update it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222715)
Time Spent: 3h (was: 2h 50m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222714
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 04/Apr/19 00:00
Start Date: 04/Apr/19 00:00
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271976703
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmVolumeOwnerChangeResponse.java
##
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.helpers;
+
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.VolumeList;
+
+/**
+ * OM response for owner change request for a ozone volume.
+ */
+public class OmVolumeOwnerChangeResponse {
+ private VolumeList originalOwnerVolumeList;
Review comment:
I think you have figured out this, but adding my response here.
Added these fields because during applyTransaction we don't want to read
OMDB, in applyTransaction, it is just applied to OM DB. (Like put call, commit
batch call). If we don't return these values, we need to read OM DB again in
applyTransaction.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222714)
Time Spent: 2h 50m (was: 2h 40m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222713
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:58
Start Date: 03/Apr/19 23:58
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271976281
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1594,6 +1598,107 @@ public void createVolume(OmVolumeArgs args) throws
IOException {
}
}
+ @Override
+ public VolumeList startCreateVolume(OmVolumeArgs args) throws IOException {
+try {
+ // TODO: Need to add metrics and Audit log for HA requests
+ if(isAclEnabled) {
+checkAcls(ResourceType.VOLUME, StoreType.OZONE,
+ACLType.CREATE, args.getVolume(), null, null);
+ }
+ VolumeList volumeList = volumeManager.createVolume(args);
+ return volumeList;
+} catch (Exception ex) {
+ throw ex;
Review comment:
Will update it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222713)
Time Spent: 2h 40m (was: 2.5h)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222712
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:58
Start Date: 03/Apr/19 23:58
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271976232
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/S3BucketManagerImpl.java
##
@@ -166,6 +170,10 @@ public boolean createOzoneVolumeIfNeeded(String userName)
.setVolume(ozoneVolumeName)
.setQuotaInBytes(OzoneConsts.MAX_QUOTA_IN_BYTES)
.build();
+ if (isRatisEnabled) {
Review comment:
S3Bucket create internally calls createVolume. As now we have not separated
s3 createbucket into 2 phase, we need to do this (In applyTransaction
createBucket is called now). As createVolume when ratis is enabled it does not
apply to OM DB. So, we need to call apply also here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222712)
Time Spent: 2.5h (was: 2h 20m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222710
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:56
Start Date: 03/Apr/19 23:56
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271975878
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java
##
@@ -363,6 +365,199 @@ public OMResponse handle(OMRequest request) {
return responseBuilder.build();
}
+ @Override
+ public OMRequest handleStartTransaction(OMRequest omRequest)
+ throws IOException {
+LOG.debug("Received OMRequest: {}, ", omRequest);
+Type cmdType = omRequest.getCmdType();
+OMRequest newOmRequest = null;
+try {
+ switch (cmdType) {
+ case CreateVolume:
+newOmRequest = handleCreateVolumeStart(omRequest);
+break;
+ case SetVolumeProperty:
+newOmRequest = handleSetVolumePropertyStart(omRequest);
+break;
+ case DeleteVolume:
+newOmRequest = handleDeleteVolumeStart(omRequest);
+break;
+ default:
+new OMException("Unrecognized Command Type:" + cmdType,
+OMException.ResultCodes.INVALID_REQUEST);
+ }
+} catch (IOException ex) {
+ throw ex;
+}
+return newOmRequest;
+ }
+
+
+ @Override
+ public OMResponse handleApplyTransaction(OMRequest omRequest) {
+LOG.debug("Received OMRequest: {}, ", omRequest);
+Type cmdType = omRequest.getCmdType();
+OMResponse.Builder responseBuilder = OMResponse.newBuilder()
+.setCmdType(cmdType)
+.setStatus(Status.OK);
+try {
+ switch (cmdType) {
+ case CreateVolume:
+responseBuilder.setCreateVolumeResponse(
+handleCreateVolumeApply(omRequest));
+break;
+ case SetVolumeProperty:
+responseBuilder.setSetVolumePropertyResponse(
+handleSetVolumePropertyApply(omRequest));
+break;
+ case DeleteVolume:
+responseBuilder.setDeleteVolumeResponse(
+handleDeleteVolumeApply(omRequest));
+break;
+ default:
+// As all request types are not changed so we need to call handle
+// here.
+return handle(omRequest);
+ }
+ responseBuilder.setSuccess(true);
+} catch (IOException ex) {
+ responseBuilder.setSuccess(false);
+ responseBuilder.setStatus(exceptionToResponseStatus(ex));
+ if (ex.getMessage() != null) {
+responseBuilder.setMessage(ex.getMessage());
+ }
+}
+return responseBuilder.build();
+ }
+
+
+ private OMRequest handleCreateVolumeStart(OMRequest omRequest)
+ throws IOException {
+try {
+ OzoneManagerProtocolProtos.VolumeInfo volumeInfo =
+ omRequest.getCreateVolumeRequest().getVolumeInfo();
+ OzoneManagerProtocolProtos.VolumeList volumeList =
+ impl.startCreateVolume(OmVolumeArgs.getFromProtobuf(volumeInfo));
+
+ CreateVolumeRequest createVolumeRequest =
+ CreateVolumeRequest.newBuilder().setVolumeInfo(volumeInfo)
+ .setVolumeList(volumeList).build();
+ return omRequest.toBuilder().setCreateVolumeRequest(createVolumeRequest)
+ .build();
+} catch (IOException ex) {
+ throw ex;
+}
+ }
+
+ private CreateVolumeResponse handleCreateVolumeApply(OMRequest omRequest)
+ throws IOException {
+try {
+ OzoneManagerProtocolProtos.VolumeInfo volumeInfo =
+ omRequest.getCreateVolumeRequest().getVolumeInfo();
+ OzoneManagerProtocolProtos.VolumeList volumeList =
+ omRequest.getCreateVolumeRequest().getVolumeList();
+ impl.applyCreateVolume(OmVolumeArgs.getFromProtobuf(volumeInfo),
+ volumeList);
+} catch (IOException ex) {
+ throw ex;
+}
+return CreateVolumeResponse.newBuilder().build();
+ }
+
+ private OMRequest handleSetVolumePropertyStart(OMRequest omRequest)
+ throws IOException {
+SetVolumePropertyRequest setVolumePropertyRequest =
+omRequest.getSetVolumePropertyRequest();
+String volume = setVolumePropertyRequest.getVolumeName();
+OMRequest newOmRequest = null;
+if (setVolumePropertyRequest.hasQuotaInBytes()) {
+ long quota = setVolumePropertyRequest.getQuotaInBytes();
+ OmVolumeArgs omVolumeArgs = impl.startSetQuota(volume, quota);
+ SetVolumePropertyRequest newSetVolumePropertyRequest =
+ SetVolumePropertyRequest.newBuilder().setVolumeName(volume)
+ .setVolumeInfo(omVolumeArgs.getProtobuf()).build();
+ newOmRequest =
+
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222709=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222709
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:55
Start Date: 03/Apr/19 23:55
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271975546
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeManagerImpl.java
##
@@ -322,28 +409,56 @@ public void deleteVolume(String volume) throws
IOException {
Preconditions.checkState(volume.equals(volumeArgs.getVolume()));
// delete the volume from the owner list
// as well as delete the volume entry
- try (BatchOperation batch = metadataManager.getStore()
- .initBatchOperation()) {
-delVolumeFromOwnerList(volume, volumeArgs.getOwnerName(), batch);
-metadataManager.getVolumeTable().deleteWithBatch(batch, dbVolumeKey);
-metadataManager.getStore().commitBatchOperation(batch);
+ VolumeList newVolumeList = delVolumeFromOwnerList(volume,
+ volumeArgs.getOwnerName());
+
+ if (!isRatisEnabled) {
+deleteVolumeCommitToDB(newVolumeList,
+volume, owner);
}
-} catch (RocksDBException| IOException ex) {
+ return new OmDeleteVolumeResponse(volume, owner, newVolumeList);
+} catch (IOException ex) {
if (!(ex instanceof OMException)) {
LOG.error("Delete volume failed for volume:{}", volume, ex);
}
- if(ex instanceof RocksDBException) {
-throw RocksDBStore.toIOException("Volume creation failed.",
-(RocksDBException) ex);
- } else {
-throw (IOException) ex;
- }
+ throw ex;
} finally {
metadataManager.getLock().releaseVolumeLock(volume);
metadataManager.getLock().releaseUserLock(owner);
}
}
+ @Override
+ public void applyDeleteVolume(String volume, String owner,
+ VolumeList newVolumeList) throws IOException {
+try {
+ deleteVolumeCommitToDB(newVolumeList, volume, owner);
+} catch (IOException ex) {
+ LOG.error("Delete volume failed for volume:{}", volume,
+ ex);
+ throw ex;
+}
+ }
+
+ private void deleteVolumeCommitToDB(VolumeList newVolumeList,
+ String volume, String owner) throws IOException {
+try (BatchOperation batch = metadataManager.getStore()
+.initBatchOperation()) {
+ String dbUserKey = metadataManager.getUserKey(owner);
Review comment:
This is not DB reading, this method returns the same user name which we have
passed. I think this was added for future purpose I believe if we want to have
a different key for user
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222709)
Time Spent: 2h 10m (was: 2h)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222708
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:55
Start Date: 03/Apr/19 23:55
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271975546
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeManagerImpl.java
##
@@ -322,28 +409,56 @@ public void deleteVolume(String volume) throws
IOException {
Preconditions.checkState(volume.equals(volumeArgs.getVolume()));
// delete the volume from the owner list
// as well as delete the volume entry
- try (BatchOperation batch = metadataManager.getStore()
- .initBatchOperation()) {
-delVolumeFromOwnerList(volume, volumeArgs.getOwnerName(), batch);
-metadataManager.getVolumeTable().deleteWithBatch(batch, dbVolumeKey);
-metadataManager.getStore().commitBatchOperation(batch);
+ VolumeList newVolumeList = delVolumeFromOwnerList(volume,
+ volumeArgs.getOwnerName());
+
+ if (!isRatisEnabled) {
+deleteVolumeCommitToDB(newVolumeList,
+volume, owner);
}
-} catch (RocksDBException| IOException ex) {
+ return new OmDeleteVolumeResponse(volume, owner, newVolumeList);
+} catch (IOException ex) {
if (!(ex instanceof OMException)) {
LOG.error("Delete volume failed for volume:{}", volume, ex);
}
- if(ex instanceof RocksDBException) {
-throw RocksDBStore.toIOException("Volume creation failed.",
-(RocksDBException) ex);
- } else {
-throw (IOException) ex;
- }
+ throw ex;
} finally {
metadataManager.getLock().releaseVolumeLock(volume);
metadataManager.getLock().releaseUserLock(owner);
}
}
+ @Override
+ public void applyDeleteVolume(String volume, String owner,
+ VolumeList newVolumeList) throws IOException {
+try {
+ deleteVolumeCommitToDB(newVolumeList, volume, owner);
+} catch (IOException ex) {
+ LOG.error("Delete volume failed for volume:{}", volume,
+ ex);
+ throw ex;
+}
+ }
+
+ private void deleteVolumeCommitToDB(VolumeList newVolumeList,
+ String volume, String owner) throws IOException {
+try (BatchOperation batch = metadataManager.getStore()
+.initBatchOperation()) {
+ String dbUserKey = metadataManager.getUserKey(owner);
Review comment:
This is not DB reading, this method applys, it returns the same user name
which we have passed. I think this was added for future purpose I believe if we
want to have a different key for user
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222708)
Time Spent: 2h (was: 1h 50m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222707
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:54
Start Date: 03/Apr/19 23:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271975546
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeManagerImpl.java
##
@@ -322,28 +409,56 @@ public void deleteVolume(String volume) throws
IOException {
Preconditions.checkState(volume.equals(volumeArgs.getVolume()));
// delete the volume from the owner list
// as well as delete the volume entry
- try (BatchOperation batch = metadataManager.getStore()
- .initBatchOperation()) {
-delVolumeFromOwnerList(volume, volumeArgs.getOwnerName(), batch);
-metadataManager.getVolumeTable().deleteWithBatch(batch, dbVolumeKey);
-metadataManager.getStore().commitBatchOperation(batch);
+ VolumeList newVolumeList = delVolumeFromOwnerList(volume,
+ volumeArgs.getOwnerName());
+
+ if (!isRatisEnabled) {
+deleteVolumeCommitToDB(newVolumeList,
+volume, owner);
}
-} catch (RocksDBException| IOException ex) {
+ return new OmDeleteVolumeResponse(volume, owner, newVolumeList);
+} catch (IOException ex) {
if (!(ex instanceof OMException)) {
LOG.error("Delete volume failed for volume:{}", volume, ex);
}
- if(ex instanceof RocksDBException) {
-throw RocksDBStore.toIOException("Volume creation failed.",
-(RocksDBException) ex);
- } else {
-throw (IOException) ex;
- }
+ throw ex;
} finally {
metadataManager.getLock().releaseVolumeLock(volume);
metadataManager.getLock().releaseUserLock(owner);
}
}
+ @Override
+ public void applyDeleteVolume(String volume, String owner,
+ VolumeList newVolumeList) throws IOException {
+try {
+ deleteVolumeCommitToDB(newVolumeList, volume, owner);
+} catch (IOException ex) {
+ LOG.error("Delete volume failed for volume:{}", volume,
+ ex);
+ throw ex;
+}
+ }
+
+ private void deleteVolumeCommitToDB(VolumeList newVolumeList,
+ String volume, String owner) throws IOException {
+try (BatchOperation batch = metadataManager.getStore()
+.initBatchOperation()) {
+ String dbUserKey = metadataManager.getUserKey(owner);
Review comment:
This is not DB reading, this method applys "/" before volume.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222707)
Time Spent: 1h 50m (was: 1h 40m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222706
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:54
Start Date: 03/Apr/19 23:54
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #689:
HDDS-1379. Convert all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271975519
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeManagerImpl.java
##
@@ -255,6 +328,19 @@ public void setQuota(String volume, long quota) throws
IOException {
}
}
+ @Override
+ public void applySetQuota(OmVolumeArgs omVolumeArgs) throws IOException {
+try {
+ String dbVolumeKey = metadataManager.getVolumeKey(
Review comment:
This is not DB reading, this method applys "/" before volume.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222706)
Time Spent: 1h 40m (was: 1.5h)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222690
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271970646
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1594,6 +1598,107 @@ public void createVolume(OmVolumeArgs args) throws
IOException {
}
}
+ @Override
+ public VolumeList startCreateVolume(OmVolumeArgs args) throws IOException {
+try {
+ // TODO: Need to add metrics and Audit log for HA requests
+ if(isAclEnabled) {
Review comment:
Nitpick: need space between if and (
This needs to be fixed in multiple places.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222690)
Time Spent: 1h (was: 50m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222695=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222695
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271971268
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeManagerImpl.java
##
@@ -322,28 +409,56 @@ public void deleteVolume(String volume) throws
IOException {
Preconditions.checkState(volume.equals(volumeArgs.getVolume()));
// delete the volume from the owner list
// as well as delete the volume entry
- try (BatchOperation batch = metadataManager.getStore()
- .initBatchOperation()) {
-delVolumeFromOwnerList(volume, volumeArgs.getOwnerName(), batch);
-metadataManager.getVolumeTable().deleteWithBatch(batch, dbVolumeKey);
-metadataManager.getStore().commitBatchOperation(batch);
+ VolumeList newVolumeList = delVolumeFromOwnerList(volume,
+ volumeArgs.getOwnerName());
+
+ if (!isRatisEnabled) {
+deleteVolumeCommitToDB(newVolumeList,
+volume, owner);
}
-} catch (RocksDBException| IOException ex) {
+ return new OmDeleteVolumeResponse(volume, owner, newVolumeList);
+} catch (IOException ex) {
if (!(ex instanceof OMException)) {
LOG.error("Delete volume failed for volume:{}", volume, ex);
}
- if(ex instanceof RocksDBException) {
-throw RocksDBStore.toIOException("Volume creation failed.",
-(RocksDBException) ex);
- } else {
-throw (IOException) ex;
- }
+ throw ex;
} finally {
metadataManager.getLock().releaseVolumeLock(volume);
metadataManager.getLock().releaseUserLock(owner);
}
}
+ @Override
+ public void applyDeleteVolume(String volume, String owner,
+ VolumeList newVolumeList) throws IOException {
+try {
+ deleteVolumeCommitToDB(newVolumeList, volume, owner);
+} catch (IOException ex) {
+ LOG.error("Delete volume failed for volume:{}", volume,
+ ex);
+ throw ex;
+}
+ }
+
+ private void deleteVolumeCommitToDB(VolumeList newVolumeList,
+ String volume, String owner) throws IOException {
+try (BatchOperation batch = metadataManager.getStore()
+.initBatchOperation()) {
+ String dbUserKey = metadataManager.getUserKey(owner);
Review comment:
Same. Can we pass the userKey from start to apply?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222695)
Time Spent: 1.5h (was: 1h 20m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14407) Fix mistakenly used of SLF4j or Log4j2 logging in DatasetVolumeChecker#checkAllVolumes
[
https://issues.apache.org/jira/browse/HDFS-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809365#comment-16809365
]
Wanqiang Ji commented on HDFS-14407:
Thank [~giovanni.fumarola] for the review.
> Fix mistakenly used of SLF4j or Log4j2 logging in
> DatasetVolumeChecker#checkAllVolumes
> --
>
> Key: HDFS-14407
> URL: https://issues.apache.org/jira/browse/HDFS-14407
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wanqiang Ji
>Assignee: Wanqiang Ji
>Priority: Minor
> Attachments: HDFS-14407.001.patch
>
>
> Wrong:
> {code:java}
> LOG.warn("checkAllVolumes timed out after {} ms" +
> maxAllowedTimeForCheckMs);
> {code}
> Correct:
> {code:java}
> LOG.warn("checkAllVolumes timed out after {} ms",
> maxAllowedTimeForCheckMs);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222693
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271970849
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/S3BucketManagerImpl.java
##
@@ -166,6 +170,10 @@ public boolean createOzoneVolumeIfNeeded(String userName)
.setVolume(ozoneVolumeName)
.setQuotaInBytes(OzoneConsts.MAX_QUOTA_IN_BYTES)
.build();
+ if (isRatisEnabled) {
Review comment:
Why do we need to call applyCreateVolume separately? Is this being pushed
through Ratis consensus correctly?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222693)
Time Spent: 1h 20m (was: 1h 10m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222692
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271971359
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java
##
@@ -363,6 +365,199 @@ public OMResponse handle(OMRequest request) {
return responseBuilder.build();
}
+ @Override
+ public OMRequest handleStartTransaction(OMRequest omRequest)
+ throws IOException {
+LOG.debug("Received OMRequest: {}, ", omRequest);
+Type cmdType = omRequest.getCmdType();
+OMRequest newOmRequest = null;
+try {
+ switch (cmdType) {
+ case CreateVolume:
+newOmRequest = handleCreateVolumeStart(omRequest);
+break;
+ case SetVolumeProperty:
+newOmRequest = handleSetVolumePropertyStart(omRequest);
+break;
+ case DeleteVolume:
+newOmRequest = handleDeleteVolumeStart(omRequest);
+break;
+ default:
+new OMException("Unrecognized Command Type:" + cmdType,
+OMException.ResultCodes.INVALID_REQUEST);
+ }
+} catch (IOException ex) {
+ throw ex;
+}
+return newOmRequest;
+ }
+
+
+ @Override
+ public OMResponse handleApplyTransaction(OMRequest omRequest) {
+LOG.debug("Received OMRequest: {}, ", omRequest);
+Type cmdType = omRequest.getCmdType();
+OMResponse.Builder responseBuilder = OMResponse.newBuilder()
+.setCmdType(cmdType)
+.setStatus(Status.OK);
+try {
+ switch (cmdType) {
+ case CreateVolume:
+responseBuilder.setCreateVolumeResponse(
+handleCreateVolumeApply(omRequest));
+break;
+ case SetVolumeProperty:
+responseBuilder.setSetVolumePropertyResponse(
+handleSetVolumePropertyApply(omRequest));
+break;
+ case DeleteVolume:
+responseBuilder.setDeleteVolumeResponse(
+handleDeleteVolumeApply(omRequest));
+break;
+ default:
+// As all request types are not changed so we need to call handle
+// here.
+return handle(omRequest);
+ }
+ responseBuilder.setSuccess(true);
+} catch (IOException ex) {
+ responseBuilder.setSuccess(false);
+ responseBuilder.setStatus(exceptionToResponseStatus(ex));
+ if (ex.getMessage() != null) {
+responseBuilder.setMessage(ex.getMessage());
+ }
+}
+return responseBuilder.build();
+ }
+
+
+ private OMRequest handleCreateVolumeStart(OMRequest omRequest)
+ throws IOException {
+try {
+ OzoneManagerProtocolProtos.VolumeInfo volumeInfo =
+ omRequest.getCreateVolumeRequest().getVolumeInfo();
+ OzoneManagerProtocolProtos.VolumeList volumeList =
+ impl.startCreateVolume(OmVolumeArgs.getFromProtobuf(volumeInfo));
+
+ CreateVolumeRequest createVolumeRequest =
+ CreateVolumeRequest.newBuilder().setVolumeInfo(volumeInfo)
+ .setVolumeList(volumeList).build();
+ return omRequest.toBuilder().setCreateVolumeRequest(createVolumeRequest)
+ .build();
+} catch (IOException ex) {
+ throw ex;
+}
+ }
+
+ private CreateVolumeResponse handleCreateVolumeApply(OMRequest omRequest)
+ throws IOException {
+try {
+ OzoneManagerProtocolProtos.VolumeInfo volumeInfo =
+ omRequest.getCreateVolumeRequest().getVolumeInfo();
+ OzoneManagerProtocolProtos.VolumeList volumeList =
+ omRequest.getCreateVolumeRequest().getVolumeList();
+ impl.applyCreateVolume(OmVolumeArgs.getFromProtobuf(volumeInfo),
+ volumeList);
+} catch (IOException ex) {
+ throw ex;
+}
+return CreateVolumeResponse.newBuilder().build();
+ }
+
+ private OMRequest handleSetVolumePropertyStart(OMRequest omRequest)
+ throws IOException {
+SetVolumePropertyRequest setVolumePropertyRequest =
+omRequest.getSetVolumePropertyRequest();
+String volume = setVolumePropertyRequest.getVolumeName();
+OMRequest newOmRequest = null;
+if (setVolumePropertyRequest.hasQuotaInBytes()) {
+ long quota = setVolumePropertyRequest.getQuotaInBytes();
+ OmVolumeArgs omVolumeArgs = impl.startSetQuota(volume, quota);
+ SetVolumePropertyRequest newSetVolumePropertyRequest =
+ SetVolumePropertyRequest.newBuilder().setVolumeName(volume)
+ .setVolumeInfo(omVolumeArgs.getProtobuf()).build();
+ newOmRequest =
+
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222691
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271970765
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1594,6 +1598,107 @@ public void createVolume(OmVolumeArgs args) throws
IOException {
}
}
+ @Override
+ public VolumeList startCreateVolume(OmVolumeArgs args) throws IOException {
+try {
+ // TODO: Need to add metrics and Audit log for HA requests
+ if(isAclEnabled) {
+checkAcls(ResourceType.VOLUME, StoreType.OZONE,
+ACLType.CREATE, args.getVolume(), null, null);
+ }
+ VolumeList volumeList = volumeManager.createVolume(args);
+ return volumeList;
+} catch (Exception ex) {
+ throw ex;
Review comment:
Don't catch and throw same exception. This is an anti-pattern. This needs to
be fixed in multiple places in different files.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222691)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222689
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271970445
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmVolumeOwnerChangeResponse.java
##
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.helpers;
+
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos
+.VolumeList;
+
+/**
+ * OM response for owner change request for a ozone volume.
+ */
+public class OmVolumeOwnerChangeResponse {
+ private VolumeList originalOwnerVolumeList;
Review comment:
Why does the response need to have all these fields? Still reviewing so it
may be clearer later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222689)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222694
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271971191
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/VolumeManagerImpl.java
##
@@ -255,6 +328,19 @@ public void setQuota(String volume, long quota) throws
IOException {
}
}
+ @Override
+ public void applySetQuota(OmVolumeArgs omVolumeArgs) throws IOException {
+try {
+ String dbVolumeKey = metadataManager.getVolumeKey(
Review comment:
We should not need to do a read in applySetQuota. Can we pass the
dbVolumeKey from start to apply?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222694)
Time Spent: 1h 20m (was: 1h 10m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[
https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222688
]
ASF GitHub Bot logged work on HDDS-1379:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:32
Start Date: 03/Apr/19 23:32
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #689: HDDS-1379. Convert
all OM Volume related operations to HA model.
URL: https://github.com/apache/hadoop/pull/689#discussion_r271970568
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##
@@ -1594,6 +1598,107 @@ public void createVolume(OmVolumeArgs args) throws
IOException {
}
}
+ @Override
+ public VolumeList startCreateVolume(OmVolumeArgs args) throws IOException {
+try {
+ // TODO: Need to add metrics and Audit log for HA requests
Review comment:
Let's file a follow up jira. Audit especially is critical to fix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222688)
Time Spent: 50m (was: 40m)
> Convert all OM Volume related operations to HA model
>
>
> Key: HDDS-1379
> URL: https://issues.apache.org/jira/browse/HDDS-1379
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In this jira, we shall convert all OM related operations to OM HA model,
> which is a 2 step.
> # StartTransaction, where we validate request and check for any errors and
> return the response.
> # ApplyTransaction, where original OM request will have a response which
> needs to be applied to OM DB. This step is just to apply response to Om DB.
> In this way, all requests which are failed with like volume not found or some
> conditions which i have not satisfied like when deleting volume should be
> empty, these all will be executed during startTransaction, and if it fails
> these requests will not be written to raft log also.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-1105) Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone Manager.
[ https://issues.apache.org/jira/browse/HDDS-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-1105 started by Aravindan Vijayan. --- > Add mechanism in Recon to obtain DB snapshot 'delta' updates from Ozone > Manager. > > > Key: HDDS-1105 > URL: https://issues.apache.org/jira/browse/HDDS-1105 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > *Some context* > The FSCK server will periodically invoke this OM API passing in the most > recent sequence number of its own RocksDB instance. The OM will use the > RockDB getUpdateSince() API to answer this query. Since the getUpdateSince > API only works against the RocksDB WAL, we have to configure OM RocksDB WAL > (https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log) with sufficient > max size to make this API useful. If the OM cannot get all transactions since > the given sequence number (due to WAL flushing), it can error out. In that > case the FSCK server can fall back to getting the entire checkpoint snapshot > implemented in HDDS-1085. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14327) Using FQDN instead of IP to access servers with DNS resolving
[ https://issues.apache.org/jira/browse/HDFS-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809353#comment-16809353 ] Hudson commented on HDFS-14327: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16347 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16347/]) HDFS-14327. Using FQDN instead of IP to access servers with DNS (inigoiri: rev 7b5b783f66f32012c00bef7593851392dd8cf2d5) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/MockDomainNameResolver.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConfiguredFailoverProxyProvider.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNSDomainNameResolver.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DomainNameResolver.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/AbstractNNFailoverProxyProvider.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java > Using FQDN instead of IP to access servers with DNS resolving > - > > Key: HDFS-14327 > URL: https://issues.apache.org/jira/browse/HDFS-14327 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14327.001.patch, HDFS-14327.002.patch > > > With [HDFS-14118|https://issues.apache.org/jira/browse/HDFS-14118], clients > can get the IP of the servers (NN/Routers) and use the IP addresses to access > the machine. This will fail in secure environment as Kerberos is using the > domain name (FQDN) in the principal so it won't recognize the IP addresses. > This task is mainly adding a reverse look up on the current basis and get the > domain name after the IP is fetched. After that clients will still use the > domain name to access the servers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14327) Using FQDN instead of IP to access servers with DNS resolving
[ https://issues.apache.org/jira/browse/HDFS-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14327: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > Using FQDN instead of IP to access servers with DNS resolving > - > > Key: HDFS-14327 > URL: https://issues.apache.org/jira/browse/HDFS-14327 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14327.001.patch, HDFS-14327.002.patch > > > With [HDFS-14118|https://issues.apache.org/jira/browse/HDFS-14118], clients > can get the IP of the servers (NN/Routers) and use the IP addresses to access > the machine. This will fail in secure environment as Kerberos is using the > domain name (FQDN) in the principal so it won't recognize the IP addresses. > This task is mainly adding a reverse look up on the current basis and get the > domain name after the IP is fetched. After that clients will still use the > domain name to access the servers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14327) Using FQDN instead of IP to access servers with DNS resolving
[ https://issues.apache.org/jira/browse/HDFS-14327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809352#comment-16809352 ] Íñigo Goiri commented on HDFS-14327: Thanks [~fengnanli] for the work. Committed to trunk. > Using FQDN instead of IP to access servers with DNS resolving > - > > Key: HDFS-14327 > URL: https://issues.apache.org/jira/browse/HDFS-14327 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14327.001.patch, HDFS-14327.002.patch > > > With [HDFS-14118|https://issues.apache.org/jira/browse/HDFS-14118], clients > can get the IP of the servers (NN/Routers) and use the IP addresses to access > the machine. This will fail in secure environment as Kerberos is using the > domain name (FQDN) in the principal so it won't recognize the IP addresses. > This task is mainly adding a reverse look up on the current basis and get the > domain name after the IP is fetched. After that clients will still use the > domain name to access the servers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1324:
-
Fix Version/s: 0.5.0
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
> elapsed: 21.781 s <<< FAILURE!
> java.lang.AssertionError: expected:<30> but was:<10>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testOMRetryProxy(TestOzoneManagerHA.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809345#comment-16809345
]
Hudson commented on HDDS-1324:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16346 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16346/])
HDDS-1324. TestOzoneManagerHA tests are flaky (#676) (bharat: rev
4e64f8d763fddacb2c326884e057523443ef136f)
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHA.java
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
> elapsed: 21.781 s <<< FAILURE!
> java.lang.AssertionError: expected:<30> but was:<10>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testOMRetryProxy(TestOzoneManagerHA.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham resolved HDDS-1324.
--
Resolution: Fixed
Thank You [~hanishakoneru] for the contribution and [~arpitagarwal] for the
review.
I have committed this to the trunk.
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
> elapsed: 21.781 s <<< FAILURE!
> java.lang.AssertionError: expected:<30> but was:<10>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testOMRetryProxy(TestOzoneManagerHA.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?focusedWorklogId=222680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222680
]
ASF GitHub Bot logged work on HDDS-1324:
Author: ASF GitHub Bot
Created on: 03/Apr/19 23:02
Start Date: 03/Apr/19 23:02
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #676:
HDDS-1324. TestOzoneManagerHA tests are flaky
URL: https://github.com/apache/hadoop/pull/676
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222680)
Time Spent: 50m (was: 40m)
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
> elapsed: 21.781 s <<< FAILURE!
> java.lang.AssertionError: expected:<30> but was:<10>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testOMRetryProxy(TestOzoneManagerHA.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222679 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 03/Apr/19 23:00 Start Date: 03/Apr/19 23:00 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651#issuecomment-479688829 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 1014 | trunk passed | | +1 | compile | 951 | trunk passed | | +1 | checkstyle | 189 | trunk passed | | -1 | mvnsite | 28 | ozone-manager in trunk failed. | | +1 | shadedclient | 1015 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | -1 | findbugs | 37 | ozone-manager in trunk failed. | | +1 | javadoc | 166 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 23 | Maven dependency ordering for patch | | -1 | mvninstall | 24 | integration-test in the patch failed. | | +1 | compile | 904 | the patch passed | | +1 | javac | 904 | the patch passed | | +1 | checkstyle | 195 | the patch passed | | +1 | mvnsite | 190 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 678 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 224 | the patch passed | | +1 | javadoc | 162 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 91 | common in the patch passed. | | +1 | unit | 47 | common in the patch passed. | | -1 | unit | 1162 | integration-test in the patch failed. | | +1 | unit | 60 | ozone-manager in the patch passed. | | +1 | asflicense | 55 | The patch does not generate ASF License warnings. | | | | 7584 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.TestMiniChaosOzoneCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-651/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/651 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux b30eca944121 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 366186d | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/5/artifact/out/branch-mvnsite-hadoop-ozone_ozone-manager.txt | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/5/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/5/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/5/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/5/testReport/ | | Max. process+thread count | 4333 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-651/5/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809343#comment-16809343
]
Hadoop QA commented on HDFS-10477:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
38s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} branch-2 passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
15s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} branch-2 passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed with JDK v1.8.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:da67579 |
| JIRA Issue | HDFS-10477 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964773/HDFS-10477.branch-2.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d2d8955079d5 4.4.0-138-generic
[jira] [Work logged] (HDDS-1324) TestOzoneManagerHA seems to be flaky
[
https://issues.apache.org/jira/browse/HDDS-1324?focusedWorklogId=222672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222672
]
ASF GitHub Bot logged work on HDDS-1324:
Author: ASF GitHub Bot
Created on: 03/Apr/19 22:42
Start Date: 03/Apr/19 22:42
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #676:
HDDS-1324. TestOzoneManagerHA tests are flaky
URL: https://github.com/apache/hadoop/pull/676#discussion_r271960508
##
File path:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHA.java
##
@@ -222,7 +221,7 @@ public void testMultipartUploadWithOneOmNodeDown() throws
Exception {
// Stop one of the ozone manager, to see when the OM leader changes
// multipart upload is happening successfully or not.
cluster.stopOzoneManager(leaderOMNodeId);
-
+Thread.sleep(NODE_FAILURE_TIMEOUT * 2);
Review comment:
Question: why this sleep is required?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222672)
Time Spent: 40m (was: 0.5h)
> TestOzoneManagerHA seems to be flaky
>
>
> Key: HDDS-1324
> URL: https://issues.apache.org/jira/browse/HDDS-1324
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Affects Versions: 0.5.0
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> TestOzoneManagerHA failed once with the following error:
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 105.931 s <<< FAILURE! - in org.apache.hadoop.ozone.om.TestOzoneManagerHA
> [ERROR] testOMRetryProxy(org.apache.hadoop.ozone.om.TestOzoneManagerHA) Time
> elapsed: 21.781 s <<< FAILURE!
> java.lang.AssertionError: expected:<30> but was:<10>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.testOMRetryProxy(TestOzoneManagerHA.java:305)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1333) OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security classes
[
https://issues.apache.org/jira/browse/HDDS-1333?focusedWorklogId=222664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222664
]
ASF GitHub Bot logged work on HDDS-1333:
Author: ASF GitHub Bot
Created on: 03/Apr/19 22:23
Start Date: 03/Apr/19 22:23
Worklog Time Spent: 10m
Work Description: xiaoyuyao commented on pull request #653: HDDS-1333.
OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security
classes
URL: https://github.com/apache/hadoop/pull/653#discussion_r271955896
##
File path: hadoop-ozone/dist/src/main/smoketest/env-compose.robot
##
@@ -13,4 +13,20 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-org.apache.hadoop.fs.ozone.OzoneFileSystem
Review comment:
Can you elaborate why this is removed/renamed?
This seems to break the MR use case where the node manager running against
o3fs got ClassNotFoundException on org.apache.hadoop.fs.ozone.OzoneFileSystem.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222664)
Time Spent: 6h (was: 5h 50m)
> OzoneFileSystem can't work with spark/hadoop2.7 because incompatible security
> classes
> -
>
> Key: HDDS-1333
> URL: https://issues.apache.org/jira/browse/HDDS-1333
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> The current ozonefs compatibility layer is broken by: HDDS-1299.
> The spark jobs (including hadoop 2.7) can't be executed any more:
> {code}
> 2019-03-25 09:50:08 INFO StateStoreCoordinatorRef:54 - Registered
> StateStoreCoordinator endpoint
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/crypto/key/KeyProviderTokenIssuer
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at
> org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:45)
> at
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:332)
> at
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
> at
> org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
> at
> org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:715)
> at
> org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:757)
> at
> org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:724)
> at org.apache.spark.examples.JavaWordCount.main(JavaWordCount.java:45)
> at
[jira] [Commented] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809314#comment-16809314
]
Íñigo Goiri commented on HDFS-13853:
Thanks [~ayushtkn], minor comments:
* Fix the check styles.
* I'm guessing the exception for {{DestinationOrder#valueOf()}} is a runtime
one? Otherwise, we could just let the exception pass.
* There is also {{Boolean#parseBoolean()}}; any advantage using that?
* Remove the extra space before the collon inRouterAdmin #676
> RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
> --
>
> Key: HDFS-13853
> URL: https://issues.apache.org/jira/browse/HDFS-13853
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Dibyendu Karmakar
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-13853-HDFS-13891-01.patch,
> HDFS-13853-HDFS-13891-02.patch, HDFS-13853-HDFS-13891-03.patch,
> HDFS-13853-HDFS-13891-04.patch, HDFS-13853-HDFS-13891-05.patch,
> HDFS-13853-HDFS-13891-06.patch
>
>
> {code:java}
> // Create a new entry
> Map destMap = new LinkedHashMap<>();
> for (String ns : nss) {
> destMap.put(ns, dest);
> }
> MountTable newEntry = MountTable.newInstance(mount, destMap);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1358) Recon Server REST API not working as expected.
[ https://issues.apache.org/jira/browse/HDDS-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809312#comment-16809312 ] Hudson commented on HDDS-1358: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16344 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16344/]) HDDS-1358 : Recon Server REST API not working as expected. (#668) (arp7: rev 8ff41d62434266d49fab65abacea5f9a20fe8172) * (edit) hadoop-ozone/ozone-recon/src/main/resources/webapps.recon.WEB-INF/web.xml * (delete) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconApplication.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/ContainerKeyService.java * (add) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconGuiceServletContextListener.java * (edit) hadoop-ozone/s3gateway/pom.xml * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconServer.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/recovery/ReconOmMetadataManagerImpl.java * (edit) hadoop-ozone/ozone-recon/pom.xml * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ContainerDBServiceProviderImpl.java * (add) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/ReconRestServletModule.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestContainerKeyService.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/KeyMetadata.java > Recon Server REST API not working as expected. > -- > > Key: HDDS-1358 > URL: https://issues.apache.org/jira/browse/HDDS-1358 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1358-000.patch, HDDS-1358-001.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Guice Jetty integration that is being used for Recon Server API layer is not > working as expected. Fixing that in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1211) Test SCMChillMode failing randomly in Jenkins run
[ https://issues.apache.org/jira/browse/HDDS-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809313#comment-16809313 ] Hudson commented on HDDS-1211: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16345 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16345/]) HDDS-1211. Test SCMChillMode failing randomly in Jenkins run (#543) (github: rev 358f7f9b992c1df63f99b4530fb32ba1e6623e51) * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestScmChillMode.java > Test SCMChillMode failing randomly in Jenkins run > - > > Key: HDDS-1211 > URL: https://issues.apache.org/jira/browse/HDDS-1211 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available, pushed-to-craterlake > Fix For: 0.5.0 > > Time Spent: 2h > Remaining Estimate: 0h > > java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) at > org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389) at > org.apache.hadoop.ozone.om.TestScmChillMode.testSCMChillMode(TestScmChillMode.java:286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14389) getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured
[
https://issues.apache.org/jira/browse/HDFS-14389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809309#comment-16809309
]
Wei-Chiu Chuang commented on HDFS-14389:
+1 I'm amazed how come this bug escaped all previous tests. Really good catch
> getAclStatus returns incorrect permissions and owner when an
> iNodeAttributeProvider is configured
> -
>
> Key: HDFS-14389
> URL: https://issues.apache.org/jira/browse/HDFS-14389
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-14389.001.patch
>
>
> With an inodeAttributeProvider configured in the namenode (eg Sentry), the
> permissions returned by a `hadoop fs -getfacl` command have an effective
> comment after them, even if the group permission bits are rwx, eg:
> {code}
> hadoop fs -ls /user/hive/warehouse/sample_08
> Found 1 items
> -rwxrwx--x+ 3 hive hive 46069 2019-03-22 00:19
> /user/hive/warehouse/sample_08/sample_08
> NOTE THE GROUP PERMISSIONS - rwx - No ACLs should get masked.
> hadoop fs -getfacl /user/hive/warehouse/sample_08/sample_08
> # file: /user/hive/warehouse/sample_08/sample_08
> # owner: hive
> # group: hive
> user::rwx
> group::---
> user:hive:rwx #effective:r--
> group:sentryDefaultAdmin:rwx#effective:r--
> user:admin:rwx #effective:r--
> group:systest:rwx #effective:r--
> group:hive:rwx #effective:r--
> mask::rwx
> other::--x
> {code}
> Note the effective comment, indicating group permissions of r-- which the ls
> output does not show.
> Usually this effective comment would downgrade the effective permissions, and
> a user with the group systest would not be able to write to the file /
> folder, but in this case that does not happen - this appears to be a display
> issue in the client.
> After some debugging, the problem is due to getAclStatus returning the
> permissions, owner and group of the underlying file in HDFS and not those
> from the inodeAttributeProvider - ie this call does not correctly use the
> attribute provider. Comparing the output with getFileStatus:
> {code}
> Permission from FileStatus: rwxrwx--x # Correct, the provider says the
> permissions are 771
> Permission from AclStatus: rw-r--r-- # Incorrect, these are the permissions
> from HDFS if the provided is disabled
> {code}
> Note that in this example, the underlying file permissions have group r-\-,
> and that is what is influencing the ACL output, making them effective r--.
> Within the namenode, the permissions are enforced correctly. The reason this
> is a CLI display issue is that AclCommand.java makes a call to getAclStatus,
> and from it, it gets the ACL list and the group permissions. Then it 'masks'
> the ACLs it displays using the returned group permission within the client.
> This is only for display purposes. FSPermissionChecker inside the Namenode is
> not impacted by this and does the correct thing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1211) Test SCMChillMode failing randomly in Jenkins run
[ https://issues.apache.org/jira/browse/HDDS-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1211: - Fix Version/s: 0.5.0 > Test SCMChillMode failing randomly in Jenkins run > - > > Key: HDDS-1211 > URL: https://issues.apache.org/jira/browse/HDDS-1211 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available, pushed-to-craterlake > Fix For: 0.5.0 > > Time Spent: 2h > Remaining Estimate: 0h > > java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) at > org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389) at > org.apache.hadoop.ozone.om.TestScmChillMode.testSCMChillMode(TestScmChillMode.java:286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1211) Test SCMChillMode failing randomly in Jenkins run
[ https://issues.apache.org/jira/browse/HDDS-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1211: - Resolution: Fixed Status: Resolved (was: Patch Available) Thank You [~arpitagarwal] for the review. I have committed this to the trunk. > Test SCMChillMode failing randomly in Jenkins run > - > > Key: HDDS-1211 > URL: https://issues.apache.org/jira/browse/HDDS-1211 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available, pushed-to-craterlake > Time Spent: 2h > Remaining Estimate: 0h > > java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) at > org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389) at > org.apache.hadoop.ozone.om.TestScmChillMode.testSCMChillMode(TestScmChillMode.java:286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1211) Test SCMChillMode failing randomly in Jenkins run
[ https://issues.apache.org/jira/browse/HDDS-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809308#comment-16809308 ] Bharat Viswanadham edited comment on HDDS-1211 at 4/3/19 10:02 PM: --- Thank You [~arpitagarwal] and [~ajayydv] for the review. I have committed this to the trunk. was (Author: bharatviswa): Thank You [~arpitagarwal] for the review. I have committed this to the trunk. > Test SCMChillMode failing randomly in Jenkins run > - > > Key: HDDS-1211 > URL: https://issues.apache.org/jira/browse/HDDS-1211 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available, pushed-to-craterlake > Time Spent: 2h > Remaining Estimate: 0h > > java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) at > org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389) at > org.apache.hadoop.ozone.om.TestScmChillMode.testSCMChillMode(TestScmChillMode.java:286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1211) Test SCMChillMode failing randomly in Jenkins run
[ https://issues.apache.org/jira/browse/HDDS-1211?focusedWorklogId=222656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222656 ] ASF GitHub Bot logged work on HDDS-1211: Author: ASF GitHub Bot Created on: 03/Apr/19 22:02 Start Date: 03/Apr/19 22:02 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #543: HDDS-1211. Test SCMChillMode failing randomly in Jenkins run URL: https://github.com/apache/hadoop/pull/543 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222656) Time Spent: 2h (was: 1h 50m) > Test SCMChillMode failing randomly in Jenkins run > - > > Key: HDDS-1211 > URL: https://issues.apache.org/jira/browse/HDDS-1211 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available, pushed-to-craterlake > Time Spent: 2h > Remaining Estimate: 0h > > java.lang.Thread.State: TIMED_WAITING at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) at > org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389) at > org.apache.hadoop.ozone.om.TestScmChillMode.testSCMChillMode(TestScmChillMode.java:286) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14413) HA Support for Dynamometer
Erik Krogen created HDFS-14413: -- Summary: HA Support for Dynamometer Key: HDFS-14413 URL: https://issues.apache.org/jira/browse/HDFS-14413 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Erik Krogen It would be nice if Dynamometer could handle spinning up a full 2 NN + 3 QJM cluster instead of just a single NN -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14411) Combine Dynamometer's SimulatedDataNodes into DataNodeCluster
Erik Krogen created HDFS-14411:
--
Summary: Combine Dynamometer's SimulatedDataNodes into
DataNodeCluster
Key: HDFS-14411
URL: https://issues.apache.org/jira/browse/HDFS-14411
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Erik Krogen
Dynamometer has a {{SimulatedDataNodes}} class, which is very similar to
{{DataNodeCluster}} but with some different functionality. It would be better
to combine the two to keep maintenance changes in a single place.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1189) Recon Aggregate DB schema and ORM
[ https://issues.apache.org/jira/browse/HDDS-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809303#comment-16809303 ] Arpit Agarwal commented on HDDS-1189: - +1 > Recon Aggregate DB schema and ORM > - > > Key: HDDS-1189 > URL: https://issues.apache.org/jira/browse/HDDS-1189 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.5.0 >Reporter: Siddharth Wagle >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1189.01.patch, HDDS-1189.02.patch, > HDDS-1189.03.patch, HDDS-1189.04.patch, HDDS-1189.05.patch, HDDS-1189.06.patch > > > _Objectives_ > - Define V1 of the db schema for recon service > - The current proposal is to use jOOQ as the ORM for SQL interaction. For two > main reasons: a) powerful DSL for querying, that abstracts out SQL dialects, > b) Allows code to schema and schema to code seamless transition, critical for > creating DDL through the code and unit testing across versions of the > application. > - Add e2e unit tests suite for Recon entities, created based on the design doc -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14412) Enable Dynamometer to use the local build of Hadoop by default
Erik Krogen created HDFS-14412: -- Summary: Enable Dynamometer to use the local build of Hadoop by default Key: HDFS-14412 URL: https://issues.apache.org/jira/browse/HDFS-14412 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Erik Krogen Currently, by default, Dynamometer will download a Hadoop tarball from the internet to use as the Hadoop version-under-test. Since it is bundled inside of Hadoop now, it would make more sense for it to use the current version of Hadoop by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1358) Recon Server REST API not working as expected.
[ https://issues.apache.org/jira/browse/HDDS-1358?focusedWorklogId=222650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222650 ] ASF GitHub Bot logged work on HDDS-1358: Author: ASF GitHub Bot Created on: 03/Apr/19 21:52 Start Date: 03/Apr/19 21:52 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #668: HDDS-1358 : Recon Server REST API not working as expected. URL: https://github.com/apache/hadoop/pull/668 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222650) Time Spent: 1h (was: 50m) > Recon Server REST API not working as expected. > -- > > Key: HDDS-1358 > URL: https://issues.apache.org/jira/browse/HDDS-1358 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1358-000.patch, HDDS-1358-001.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Guice Jetty integration that is being used for Recon Server API layer is not > working as expected. Fixing that in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14410) Make Dynamometer documentation properly compile onto the Hadoop site
Erik Krogen created HDFS-14410: -- Summary: Make Dynamometer documentation properly compile onto the Hadoop site Key: HDFS-14410 URL: https://issues.apache.org/jira/browse/HDFS-14410 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Erik Krogen The documentation included with Dynamometer doesn't properly appear on the site, we need to twiddle with this a bit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1358) Recon Server REST API not working as expected.
[ https://issues.apache.org/jira/browse/HDDS-1358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1358: Resolution: Fixed Status: Resolved (was: Patch Available) I've committed this via GitHub. Thanks for the contribution [~avijayan]! > Recon Server REST API not working as expected. > -- > > Key: HDDS-1358 > URL: https://issues.apache.org/jira/browse/HDDS-1358 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1358-000.patch, HDDS-1358-001.patch > > Time Spent: 1h > Remaining Estimate: 0h > > Guice Jetty integration that is being used for Recon Server API layer is not > working as expected. Fixing that in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1358) Recon Server REST API not working as expected.
[ https://issues.apache.org/jira/browse/HDDS-1358?focusedWorklogId=222649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222649 ] ASF GitHub Bot logged work on HDDS-1358: Author: ASF GitHub Bot Created on: 03/Apr/19 21:51 Start Date: 03/Apr/19 21:51 Worklog Time Spent: 10m Work Description: arp7 commented on issue #668: HDDS-1358 : Recon Server REST API not working as expected. URL: https://github.com/apache/hadoop/pull/668#issuecomment-479672277 +1 lgtm. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222649) Time Spent: 50m (was: 40m) > Recon Server REST API not working as expected. > -- > > Key: HDDS-1358 > URL: https://issues.apache.org/jira/browse/HDDS-1358 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Critical > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1358-000.patch, HDDS-1358-001.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Guice Jetty integration that is being used for Recon Server API layer is not > working as expected. Fixing that in this JIRA. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14409) Improve Dynamometer test suite
Erik Krogen created HDFS-14409: -- Summary: Improve Dynamometer test suite Key: HDFS-14409 URL: https://issues.apache.org/jira/browse/HDFS-14409 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Erik Krogen The testing within Dynamometer now is mostly one big integration test. It could really use better testing throughout. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12345) Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
[
https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809299#comment-16809299
]
Erik Krogen commented on HDFS-12345:
I'm attaching a v006 patch which clears up the remaining javac warnings, fixes
the shelldocs and javadocs issues, fixes most of the remaining checkstyle
warnings, fixes most of the shellcheck issues, fixes whitespace issues, removes
a few usages of {{commons.logging}} in favor of {{slf4j}}, removes use of
private YARN APIs within {{ApplicationMaster}}, and puts a timeout on the main
integration test.
I considered trying to fold {{SimulatedDataNodes}} into {{DataNodeCluster}},
but I think I would prefer to leave it as a follow-on item. For now, this patch
is completely isolated to new code within the
{{hadoop-tools/hadoop-dynamometer}} submodule (besides a few changes to build
and assembly files), and I would prefer to keep it that way. Merging SDN into
DNC should remove the last of the burden of private API usage.
Two things I noticed while going through this pass that I would like to discuss:
* This currently includes a class to allow Dynamometer to be run via Azkaban.
This is very useful for us internally, but may not make much sense to have
within Hadoop. Unless I hear another opinion, I will probably remove this in a
subsequent patch.
* Currently, {{TestDynamometerInfra}} requires {{bash}} and {{tar}} to be
present on the PATH, so it will fail on Windows. I'd like to remove the
reliance on {{tar}} completely, but I think it will be a bit more tricky, so I
think it would be reasonable to simply disable building Dynamometer on Windows
for now. I'm curious if anyone knows how to achieve this -- [~smeng] ? It seems
it should be possible through some POM magic.
> Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
> --
>
> Key: HDFS-12345
> URL: https://issues.apache.org/jira/browse/HDFS-12345
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: namenode, test
>Reporter: Zhe Zhang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-12345.000.patch, HDFS-12345.001.patch,
> HDFS-12345.002.patch, HDFS-12345.003.patch, HDFS-12345.004.patch,
> HDFS-12345.005.patch, HDFS-12345.006.patch
>
>
> Dynamometer has now been open sourced on our [GitHub
> page|https://github.com/linkedin/dynamometer]. Read more at our [recent blog
> post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum].
> To encourage getting the tool into the open for others to use as quickly as
> possible, we went through our standard open sourcing process of releasing on
> GitHub. However we are interested in the possibility of donating this to
> Apache as part of Hadoop itself and would appreciate feedback on whether or
> not this is something that would be supported by the community.
> Also of note, previous [discussions on the dev mail
> lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%3c98fceffa-faff-4cf1-a14d-4faab6567...@gmail.com%3e]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12345) Scale testing HDFS NameNode with real metadata and workloads (Dynamometer)
[ https://issues.apache.org/jira/browse/HDFS-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-12345: --- Attachment: HDFS-12345.006.patch > Scale testing HDFS NameNode with real metadata and workloads (Dynamometer) > -- > > Key: HDFS-12345 > URL: https://issues.apache.org/jira/browse/HDFS-12345 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode, test >Reporter: Zhe Zhang >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-12345.000.patch, HDFS-12345.001.patch, > HDFS-12345.002.patch, HDFS-12345.003.patch, HDFS-12345.004.patch, > HDFS-12345.005.patch, HDFS-12345.006.patch > > > Dynamometer has now been open sourced on our [GitHub > page|https://github.com/linkedin/dynamometer]. Read more at our [recent blog > post|https://engineering.linkedin.com/blog/2018/02/dynamometer--scale-testing-hdfs-on-minimal-hardware-with-maximum]. > To encourage getting the tool into the open for others to use as quickly as > possible, we went through our standard open sourcing process of releasing on > GitHub. However we are interested in the possibility of donating this to > Apache as part of Hadoop itself and would appreciate feedback on whether or > not this is something that would be supported by the community. > Also of note, previous [discussions on the dev mail > lists|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201707.mbox/%3c98fceffa-faff-4cf1-a14d-4faab6567...@gmail.com%3e] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13853) RBF: RouterAdmin update cmd is overwriting the entry not updating the existing
[
https://issues.apache.org/jira/browse/HDFS-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809288#comment-16809288
]
Hadoop QA commented on HDFS-13853:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
19s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
14s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-13891 passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
55s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
33s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13853 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964735/HDFS-13853-HDFS-13891-06.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 54d426ce8c5c 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-14390) Provide kerberos support for AliasMap service used by Provided storage
[
https://issues.apache.org/jira/browse/HDFS-14390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809281#comment-16809281
]
Ashvin commented on HDFS-14390:
---
[~elgoiri] [~virajith], thanks for the review.
Hi [~jlowe] [~crh], [~subru] and [~elgoiri] mentioned that your feedback would
be valuable. Could you please take a look at the changes. Thanks !
> Provide kerberos support for AliasMap service used by Provided storage
> --
>
> Key: HDFS-14390
> URL: https://issues.apache.org/jira/browse/HDFS-14390
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ashvin
>Assignee: Ashvin
>Priority: Major
> Attachments: HDFS-14390.001.patch, HDFS-14390.002.patch,
> HDFS-14390.003.patch
>
>
> With {{PROVIDED}} storage (-HDFS-9806)-, HDFS can address data stored in
> external storage systems. This feature is not supported in a secure HDFS
> cluster. The {{AliasMap}} service does not support kerberos, and as a result
> the cluster nodes will fail to communicate with it. This JIRA is to enable
> kerberos support for the {{AliasMap}} service.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1353) Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing because of BackgroundPipelineCreator
[
https://issues.apache.org/jira/browse/HDDS-1353?focusedWorklogId=222607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222607
]
ASF GitHub Bot logged work on HDDS-1353:
Author: ASF GitHub Bot
Created on: 03/Apr/19 20:38
Start Date: 03/Apr/19 20:38
Worklog Time Spent: 10m
Work Description: vivekratnavel commented on issue #681: HDDS-1353 :
Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing
because of BackgroundPipelineCreator.
URL: https://github.com/apache/hadoop/pull/681#issuecomment-479649552
+1 LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 222607)
Time Spent: 50m (was: 40m)
> Metrics scm_pipeline_metrics_num_pipeline_creation_failed keeps increasing
> because of BackgroundPipelineCreator
> ---
>
> Key: HDDS-1353
> URL: https://issues.apache.org/jira/browse/HDDS-1353
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Aravindan Vijayan
>Priority: Minor
> Labels: newbie, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> There is a {{BackgroundPipelineCreator}} thread in SCM which runs in a fixed
> interval and tries to create pipelines. This BackgroundPipelineCreator uses
> {{IOException}} as exit criteria (no more pipelines can be created). In each
> run of BackgroundPipelineCreator we exit when we are not able to create any
> more pipelines, i.e. when we get IOException while trying to create the
> pipeline. This means that
> {{scm_pipeline_metrics_num_pipeline_creation_failed}} value will get
> incremented in each run of BackgroundPipelineCreator.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809235#comment-16809235
]
Wei-Chiu Chuang edited comment on HDFS-10477 at 4/3/19 8:36 PM:
Here's the branch-2 patch. It partially includes HDFS-13027, and I think it
makes sense to backport HDFS-13027 to branch-2 also.
[^HDFS-10477.branch-2.patch]
was (Author: jojochuang):
Here's the branch-2 patch. It partially includes HDFS-13027, and I think it
makes sense to backport HDFS-13027 to branch-2 also.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.branch-2.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809235#comment-16809235
]
Wei-Chiu Chuang commented on HDFS-10477:
Here's the branch-2 patch. It partially includes HDFS-13027, and I think it
makes sense to backport HDFS-13027 to branch-2 also.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.branch-2.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
[jira] [Updated] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10477:
---
Attachment: HDFS-10477.branch-2.patch
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.branch-2.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004
[jira] [Work logged] (HDDS-1379) Convert all OM Volume related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1379?focusedWorklogId=222606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222606 ] ASF GitHub Bot logged work on HDDS-1379: Author: ASF GitHub Bot Created on: 03/Apr/19 20:30 Start Date: 03/Apr/19 20:30 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #689: HDDS-1379. Convert all OM Volume related operations to HA model. URL: https://github.com/apache/hadoop/pull/689#issuecomment-479647097 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 26 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 63 | Maven dependency ordering for branch | | +1 | mvninstall | 1034 | trunk passed | | +1 | compile | 98 | trunk passed | | +1 | checkstyle | 26 | trunk passed | | +1 | mvnsite | 97 | trunk passed | | +1 | shadedclient | 750 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 93 | trunk passed | | +1 | javadoc | 68 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 10 | Maven dependency ordering for patch | | +1 | mvninstall | 97 | the patch passed | | +1 | compile | 88 | the patch passed | | +1 | cc | 88 | the patch passed | | +1 | javac | 88 | the patch passed | | -0 | checkstyle | 22 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 79 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 706 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | -1 | findbugs | 50 | hadoop-ozone/ozone-manager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | -1 | javadoc | 35 | hadoop-ozone_common generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) | ||| _ Other Tests _ | | +1 | unit | 36 | common in the patch passed. | | +1 | unit | 42 | ozone-manager in the patch passed. | | -1 | unit | 784 | integration-test in the patch failed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 4380 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-ozone/ozone-manager | | | Dead store to volume in org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleSetVolumePropertyApply(OzoneManagerProtocolProtos$OMRequest) At OzoneManagerRequestHandler.java:org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleSetVolumePropertyApply(OzoneManagerProtocolProtos$OMRequest) At OzoneManagerRequestHandler.java:[line 512] | | | new org.apache.hadoop.ozone.om.exceptions.OMException(String, OMException$ResultCodes) not thrown in org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleStartTransaction(OzoneManagerProtocolProtos$OMRequest) At OzoneManagerRequestHandler.java:in org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleStartTransaction(OzoneManagerProtocolProtos$OMRequest) At OzoneManagerRequestHandler.java:[line 386] | | Failed junit tests | hadoop.ozone.om.TestOmMetrics | | | hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean | | | hadoop.ozone.TestMiniChaosOzoneCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-689/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/689 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 2fe446930f83 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d797907 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-689/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-689/1/artifact/out/new-findbugs-hadoop-ozone_ozone-manager.html | | javadoc |
[jira] [Commented] (HDDS-1330) Add a docker compose for Ozone deployment with Recon.
[ https://issues.apache.org/jira/browse/HDDS-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809224#comment-16809224 ] Hudson commented on HDDS-1330: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16341 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16341/]) HDDS-1330 : Add a docker compose for Ozone deployment with Recon. (#669) (arp7: rev 59022b28b7cca1d9b5867bc0787d0b5d0ad1a3e7) * (add) hadoop-ozone/dist/src/main/compose/ozone-recon/docker-compose.yaml * (add) hadoop-ozone/dist/src/main/compose/ozone-recon/.env * (add) hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config > Add a docker compose for Ozone deployment with Recon. > - > > Key: HDDS-1330 > URL: https://issues.apache.org/jira/browse/HDDS-1330 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1330-000.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Add a docker compose for Ozone deployment with Recon. > * Test out Recon container key service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1330) Add a docker compose for Ozone deployment with Recon.
[ https://issues.apache.org/jira/browse/HDDS-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDDS-1330: Resolution: Fixed Status: Resolved (was: Patch Available) I've committed this. Thanks for the contribution [~avijayan]! > Add a docker compose for Ozone deployment with Recon. > - > > Key: HDDS-1330 > URL: https://issues.apache.org/jira/browse/HDDS-1330 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1330-000.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Add a docker compose for Ozone deployment with Recon. > * Test out Recon container key service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1330) Add a docker compose for Ozone deployment with Recon.
[ https://issues.apache.org/jira/browse/HDDS-1330?focusedWorklogId=222603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222603 ] ASF GitHub Bot logged work on HDDS-1330: Author: ASF GitHub Bot Created on: 03/Apr/19 20:20 Start Date: 03/Apr/19 20:20 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #669: HDDS-1330 : Add a docker compose for Ozone deployment with Recon. URL: https://github.com/apache/hadoop/pull/669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222603) Time Spent: 50m (was: 40m) > Add a docker compose for Ozone deployment with Recon. > - > > Key: HDDS-1330 > URL: https://issues.apache.org/jira/browse/HDDS-1330 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1330-000.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Add a docker compose for Ozone deployment with Recon. > * Test out Recon container key service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1330) Add a docker compose for Ozone deployment with Recon.
[ https://issues.apache.org/jira/browse/HDDS-1330?focusedWorklogId=222602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222602 ] ASF GitHub Bot logged work on HDDS-1330: Author: ASF GitHub Bot Created on: 03/Apr/19 20:20 Start Date: 03/Apr/19 20:20 Worklog Time Spent: 10m Work Description: arp7 commented on issue #669: HDDS-1330 : Add a docker compose for Ozone deployment with Recon. URL: https://github.com/apache/hadoop/pull/669#issuecomment-479643721 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222602) Time Spent: 40m (was: 0.5h) > Add a docker compose for Ozone deployment with Recon. > - > > Key: HDDS-1330 > URL: https://issues.apache.org/jira/browse/HDDS-1330 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Attachments: HDDS-1330-000.patch > > Time Spent: 40m > Remaining Estimate: 0h > > * Add a docker compose for Ozone deployment with Recon. > * Test out Recon container key service. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222601 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 03/Apr/19 20:20 Start Date: 03/Apr/19 20:20 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651#issuecomment-479643547 +1 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222601) Time Spent: 4h 10m (was: 4h) > Implement Ratis Snapshots on OM > --- > > Key: HDDS-1339 > URL: https://issues.apache.org/jira/browse/HDDS-1339 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > For bootstrapping and restarting OMs, we need to implement snapshots in OM. > The OM state maintained by RocksDB will be checkpoint-ed on demand. Ratis > snapshots will only preserve the last applied log index by the State Machine > on disk. This index will be stored in file in the OM metadata dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222599=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222599 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 03/Apr/19 20:19 Start Date: 03/Apr/19 20:19 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651#discussion_r271915226 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -1617,7 +1617,7 @@ ozone.om.ratis.snapshot.auto.trigger.threshold -40L +40 Review comment: Thanks for the info. We can tweak this later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222599) Time Spent: 4h (was: 3h 50m) > Implement Ratis Snapshots on OM > --- > > Key: HDDS-1339 > URL: https://issues.apache.org/jira/browse/HDDS-1339 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > For bootstrapping and restarting OMs, we need to implement snapshots in OM. > The OM state maintained by RocksDB will be checkpoint-ed on demand. Ratis > snapshots will only preserve the last applied log index by the State Machine > on disk. This index will be stored in file in the OM metadata dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1339) Implement Ratis Snapshots on OM
[ https://issues.apache.org/jira/browse/HDDS-1339?focusedWorklogId=222597=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-222597 ] ASF GitHub Bot logged work on HDDS-1339: Author: ASF GitHub Bot Created on: 03/Apr/19 20:17 Start Date: 03/Apr/19 20:17 Worklog Time Spent: 10m Work Description: hanishakoneru commented on pull request #651: HDDS-1339. Implement ratis snapshots on OM URL: https://github.com/apache/hadoop/pull/651#discussion_r271914300 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -1617,7 +1617,7 @@ ozone.om.ratis.snapshot.auto.trigger.threshold -40L +40 Review comment: I think 400k should not be too small a number. In HDFS, the default number of transactions after which a checkpoint is saved is 1M. Also, the ratis log index is not the same as the actual transaction count. There are lot of internal ratis log entries also. But we can re-tweak the default after some testing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 222597) Time Spent: 3h 50m (was: 3h 40m) > Implement Ratis Snapshots on OM > --- > > Key: HDDS-1339 > URL: https://issues.apache.org/jira/browse/HDDS-1339 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > For bootstrapping and restarting OMs, we need to implement snapshots in OM. > The OM state maintained by RocksDB will be checkpoint-ed on demand. Ratis > snapshots will only preserve the last applied log index by the State Machine > on disk. This index will be stored in file in the OM metadata dir. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1367) Add ability in Recon to track the growth rate of the cluster.
[ https://issues.apache.org/jira/browse/HDDS-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian reassigned HDDS-1367: Assignee: Vivek Ratnavel Subramanian > Add ability in Recon to track the growth rate of the cluster. > -- > > Key: HDDS-1367 > URL: https://issues.apache.org/jira/browse/HDDS-1367 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Fix For: 0.5.0 > > > Recon should be able to answer the question "How fast is the cluster growing, > by week, by month, by day?", which gives the user an idea of the usage stats > of the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809153#comment-16809153
]
Wei-Chiu Chuang commented on HDFS-10477:
Pushed to trunk, branch-3.2, branch-3.1 and branch-3.0.
There are some conflicts for a branch-2 commit so I'm working on that.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
>
