[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1147: HDDS-3892. Datanode initialization is too slow when there are thousan…
ChenSammi commented on pull request #1147: URL: https://github.com/apache/hadoop-ozone/pull/1147#issuecomment-652176025 > > > I see one issue with this approach. > If the database is already opened, and if we try to open again we will get this error. > > I think, with this change, we will throw an exception if we try to open the database again an already existing one. > > java.io.IOException: Failed init RocksDB, db path : /Users/bviswanadham/workspace/hadoop-ozone/hadoop-hdds/container-service/target/test-dir/xCkBnsLVrc/cont1, exception :org.rocksdb.RocksDBException lock : /Users/bviswanadham/workspace/hadoop-ozone/hadoop-hdds/container-service/target/test-dir/xCkBnsLVrc/cont1/LOCK: No locks available @bharatviswa504 , I get your point. It's an issue, but not introduced by this patch. It's a currenlty existing issue and we need to carefully think about how to fix it with a new JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1143: HDDS-3891. Add the usage of ofs in doc.
captainzmc commented on pull request #1143: URL: https://github.com/apache/hadoop-ozone/pull/1143#issuecomment-652175135 Thanks @xiaoyuyao for the review. I have fixed doc. Please take another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1143: HDDS-3891. Add the usage of ofs in doc.
captainzmc commented on a change in pull request #1143: URL: https://github.com/apache/hadoop-ozone/pull/1143#discussion_r448095312 ## File path: hadoop-hdds/docs/content/interface/OzoneFS.md ## @@ -113,3 +113,50 @@ hdfs dfs -ls o3fs://bucket.volume.om-host.example.com:6789/key Note: Only port number from the config is used in this case, whereas the host name in the config `ozone.om.address` is ignored. +## Setting up the ofs +Please add the following entry to the core-site.xml. Review comment: References to ofs.md have been added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1143: HDDS-3891. Add the usage of ofs in doc.
captainzmc commented on a change in pull request #1143: URL: https://github.com/apache/hadoop-ozone/pull/1143#discussion_r448094949 ## File path: hadoop-hdds/docs/content/interface/OzoneFS.md ## @@ -23,9 +23,9 @@ summary: Hadoop Compatible file system allows any application that expects an HD The Hadoop compatible file system interface allows storage backends like Ozone to be easily integrated into Hadoop eco-system. Ozone file system is an -Hadoop compatible file system. +Hadoop compatible file system. Currently, Ozone supports two scheme: o3fs and ofs. Review comment: The main difference between the two has been added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3905) Show status of OM in the OM web ui
[ https://issues.apache.org/jira/browse/HDDS-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3905: - Labels: pull-request-available (was: ) > Show status of OM in the OM web ui > -- > > Key: HDDS-3905 > URL: https://issues.apache.org/jira/browse/HDDS-3905 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1152: HDDS-3905. Show status of OM in the OM web ui
maobaolong opened a new pull request #1152: URL: https://github.com/apache/hadoop-ozone/pull/1152 ## What changes were proposed in this pull request? Add a Status table for OM, after this PR, we can bring more status item to the status table. ## What is the link to the Apache JIRA HDDS-3905. ## How was this patch tested?  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3191) Switch current pipeline interface to the new Replication based interface to write to Ratis
[ https://issues.apache.org/jira/browse/HDDS-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3191: - Labels: pull-request-available (was: ) > Switch current pipeline interface to the new Replication based interface to > write to Ratis > -- > > Key: HDDS-3191 > URL: https://issues.apache.org/jira/browse/HDDS-3191 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Li Cheng >Assignee: Glen Geng >Priority: Major > Labels: pull-request-available > > Due to consistency concern, SCM needs to applyTransaction to RaftLog before > it writes to local database and in memory maps. Need refactor the current > codes to put this part to Ratis. > Ratis will write to DB on behalf of SCM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng opened a new pull request #1151: HDDS-3191: switch from SCMPipelineManager to PipelineManagerV2Impl
GlenGeng opened a new pull request #1151: URL: https://github.com/apache/hadoop-ozone/pull/1151 ## What changes were proposed in this pull request? switch from SCMPipelineManager to PipelineManagerV2Impl. mainly focus on fixing unit test ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3191 ## How was this patch tested? unit tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3905) Show status of OM in the OM web ui
[ https://issues.apache.org/jira/browse/HDDS-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong reassigned HDDS-3905: Assignee: maobaolong > Show status of OM in the OM web ui > -- > > Key: HDDS-3905 > URL: https://issues.apache.org/jira/browse/HDDS-3905 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Manager >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3905) Show status of OM in the OM web ui
maobaolong created HDDS-3905: Summary: Show status of OM in the OM web ui Key: HDDS-3905 URL: https://issues.apache.org/jira/browse/HDDS-3905 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: Ozone Manager Affects Versions: 0.7.0 Reporter: maobaolong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1139: HDDS-3885. Create Datanode home page
maobaolong commented on pull request #1139: URL: https://github.com/apache/hadoop-ozone/pull/1139#issuecomment-652122430 @adoroszlai Thank you for your review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1023: HDDS-3718: Improve OmKeyLocationInfoGroup internal data structure
xiaoyuyao commented on a change in pull request #1023:
URL: https://github.com/apache/hadoop-ozone/pull/1023#discussion_r448011359
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyLocationInfoGroup.java
##
@@ -30,12 +30,23 @@
*/
public class OmKeyLocationInfoGroup {
private final long version;
- private final List locationList;
+ private final Map> locationVersionList;
Review comment:
NIT: should we rename to locationVersionMap to avoid confusion?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1023: HDDS-3718: Improve OmKeyLocationInfoGroup internal data structure
xiaoyuyao commented on a change in pull request #1023:
URL: https://github.com/apache/hadoop-ozone/pull/1023#discussion_r448010316
##
File path:
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyLocationInfoGroup.java
##
@@ -30,12 +30,23 @@
*/
public class OmKeyLocationInfoGroup {
private final long version;
- private final List locationList;
+ private final Map> locationVersionList;
public OmKeyLocationInfoGroup(long version,
List locations) {
this.version = version;
-this.locationList = locations;
+this.locationVersionList = locations.stream()
+.collect(Collectors.groupingBy(OmKeyLocationInfo::getCreateVersion));
+//prevent NPE
+this.locationVersionList.putIfAbsent(version, new ArrayList<>());
+ }
+
+ public OmKeyLocationInfoGroup(long version,
+Map> locations) {
+this.version = version;
+this.locationVersionList = locations;
Review comment:
should we change the order of 47/49 so that there is always an ArrayList
for the version and the locations is added to that arraylist.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1106: HDDS-2413. Set configuration variables from annotated java objects
elek commented on a change in pull request #1106: URL: https://github.com/apache/hadoop-ozone/pull/1106#discussion_r447520786 ## File path: hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationSource.java ## @@ -37,9 +39,6 @@ char[] getPassword(String key) throws IOException; - @Deprecated Review comment: I really don't like the approach when the current port number (after using :0) is added to the configuration. It's easier if the configuration is immutable after loading. I understand that the immutable is not possible with this injection and I accept that it's necessary. One possible approach is to introduce two interfaces: `ConfigSource` (read-only) and `ConfigDestination` (write only). `OzoneConfiguration` can implement both. But there could be a specific `ConfigSource` (eg. `EnvironmentVariableConfigSource`) which would be read only. But I am fine with committing this patch as is, but interested about opinion. Also: it seems to be useful for testing to introduce an `OzoneConfiguration.fromAnnotatedObject()` method. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3705) [OFS] Implement getTrashRoots for trash cleanup
[
https://issues.apache.org/jira/browse/HDDS-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-3705:
-
Description:
We need to override {{getTrashRoots()}} as well in order to allow for easier
future OM trash cleanup impl.
This jira doesn't directly implement the trash cleanup feature itself, but a
prerequisite for this feature.
This is a follow-up jira to HDDS-3574:
https://github.com/apache/hadoop-ozone/pull/941#discussion_r428212741
was:
We need to override {{getTrashRoots()}} as well in order to allow for easier
future OM trash cleanup impl.
This jira doesn't directly implement the trash cleanup feature itself, but a
prerequisite for this feature.
This is a follow-up jira to HDDS-3574:
https://github.com/apache/hadoop-ozone/pull/941#discussion_r428212741
May raise PR after vote for OFS merge to master
Dev commit:
https://github.com/smengcl/hadoop-ozone/commit/c563260011aa00a36568c57cda7254e82c15c59c
(Code finished. Untested. Pending unit test.)
> [OFS] Implement getTrashRoots for trash cleanup
> ---
>
> Key: HDDS-3705
> URL: https://issues.apache.org/jira/browse/HDDS-3705
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Filesystem
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Blocker
> Labels: pull-request-available
>
> We need to override {{getTrashRoots()}} as well in order to allow for easier
> future OM trash cleanup impl.
> This jira doesn't directly implement the trash cleanup feature itself, but a
> prerequisite for this feature.
> This is a follow-up jira to HDDS-3574:
> https://github.com/apache/hadoop-ozone/pull/941#discussion_r428212741
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1129: HDDS-3741. Reload old OM state if Install Snapshot from Leader fails
hanishakoneru commented on pull request #1129: URL: https://github.com/apache/hadoop-ozone/pull/1129#issuecomment-651991144 Thanks @arp7 for the review. Addressed your review comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl commented on pull request #1089: [WIP] HDDS-3705. [OFS] Implement getTrashRoots for trash cleanup
smengcl commented on pull request #1089: URL: https://github.com/apache/hadoop-ozone/pull/1089#issuecomment-651985170 I'm rebasing the branch since the last update is a while ago. Also there were some flaky ozone-recon robot test in the [first run](https://github.com/apache/hadoop-ozone/runs/782209210). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3868) Implement getTrashRoot and getTrashRoots in o3fs
[
https://issues.apache.org/jira/browse/HDDS-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-3868:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Implement getTrashRoot and getTrashRoots in o3fs
>
>
> Key: HDDS-3868
> URL: https://issues.apache.org/jira/browse/HDDS-3868
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: Ozone Filesystem
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Blocker
> Labels: pull-request-available
>
> Override the default Hadoop FS impl, which moves trash under
> {{o3fs://bucketName.volumeName.om/user/userName/.Trash/Current/...}}
> New trash location will be:
> {{o3fs://bucketName.volumeName.om/.Trash/userName/Current/...}}
> This change also unifies the trash root with OFS.
> CC [~arp] [~msingh] [~shashikant] [~sadanand_shenoy]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3803) [OFS] Add User Guide
[
https://issues.apache.org/jira/browse/HDDS-3803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-3803:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
> [OFS] Add User Guide
>
>
> Key: HDDS-3803
> URL: https://issues.apache.org/jira/browse/HDDS-3803
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Blocker
> Labels: pull-request-available
>
> Need to add a user guide markdown for OFS. Especially the usage for {{/tmp}}.
> Thanks [~umamaheswararao] and [~xyao] for the reminder.
> {{hadoop-hdds/docs/content/design/ofs.md}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] smengcl merged pull request #1144: HDDS-3803. [OFS] Add User Guide
smengcl merged pull request #1144: URL: https://github.com/apache/hadoop-ozone/pull/1144 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-3894) Noisy log at OM when the requested sequence is not written into DB
[
https://issues.apache.org/jira/browse/HDDS-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148855#comment-17148855
]
Aravindan Vijayan edited comment on HDDS-3894 at 6/30/20, 5:48 PM:
---
[~rakeshr] Thanks for tracking this issue. Since Recon can recover from this,
and the frequency of the getUpdatesSince call should be around ~10mins, we may
be ok with just suppressing the logs here. But we should understand why this is
happening.
Can you share more details on the setup here?
* Can we check if Recon was talking to the leader/follower OM? Was there a
failover before these logs started appearing?
* What is the load on OzoneManager when this happens? Are we running a stress
test on OM RocksDB?
was (Author: avijayan):
[~rakeshr] Thanks for tracking this issue. Since Recon can recover from this,
and the frequency of the getUpdatesSince call should be around ~10mins, we may
be ok with just suppressing the logs here. But we should understand why this is
happening.
Can you share more details on the setup here?
* Is OM HA enabled? Can we check if Recon was talking to the leader/follower OM?
* What is the load on OzoneManager when this happens? Are we running a stress
test on OM RocksDB?
> Noisy log at OM when the requested sequence is not written into DB
> --
>
> Key: HDDS-3894
> URL: https://issues.apache.org/jira/browse/HDDS-3894
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager, Ozone Recon
>Reporter: Rakesh Radhakrishnan
>Assignee: Rakesh Radhakrishnan
>Priority: Minor
>
> Too many logs at OM, which is noisy. It looks like Recon is making this call
> to OM to update the delta info. OM can ignore this exception and reduce the
> log priority to DEBUG or so.
> *Ozone-om.log*
> {code:java}
> 2020-06-26 10:02:52,963 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolled log segment from
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_106536
> to
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_106536-106603
> 2020-06-26 10:02:52,982 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> created new log segment
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_106604
> 2020-06-26 10:09:54,828 ERROR org.apache.hadoop.hdds.utils.db.RDBStore:
> Unable to get delta updates since sequenceNumber 11268864
> org.rocksdb.RocksDBException: Requested sequence not yet written in the db
> at org.rocksdb.RocksDB.getUpdatesSince(Native Method)
> at org.rocksdb.RocksDB.getUpdatesSince(RocksDB.java:3588)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.getUpdatesSince(RDBStore.java:339)
> at
> org.apache.hadoop.ozone.om.OzoneManager.getDBUpdates(OzoneManager.java:3422)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getOMDBUpdates(OzoneManagerRequestHandler.java:257)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:194)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:171)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:109)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:74)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:99)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
>
[jira] [Resolved] (HDDS-1261) Add robot tests for Recon Server
[ https://issues.apache.org/jira/browse/HDDS-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-1261. - Resolution: Fixed Fixed through the patch for HDDS-3117. > Add robot tests for Recon Server > > > Key: HDDS-1261 > URL: https://issues.apache.org/jira/browse/HDDS-1261 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > * Add robot tests for Recon Server to work with OM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1261) Add robot tests for Recon Server
[ https://issues.apache.org/jira/browse/HDDS-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-1261: --- Assignee: Aravindan Vijayan (was: Vivek Ratnavel Subramanian) > Add robot tests for Recon Server > > > Key: HDDS-1261 > URL: https://issues.apache.org/jira/browse/HDDS-1261 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > * Add robot tests for Recon Server to work with OM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3832) Enhance Recon UI to display the different unhealthy container states
[ https://issues.apache.org/jira/browse/HDDS-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-3832: --- Assignee: Vivek Ratnavel Subramanian > Enhance Recon UI to display the different unhealthy container states > > > Key: HDDS-3832 > URL: https://issues.apache.org/jira/browse/HDDS-3832 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Recon >Affects Versions: 0.6.0 >Reporter: Stephen O'Donnell >Assignee: Vivek Ratnavel Subramanian >Priority: Major > > When HDDS-3831 is closed, we should enhance the Recon UI to display the > different unhealthy container counts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-3873) Intermittent failure in Recon acceptance test due to too many pipelines
[
https://issues.apache.org/jira/browse/HDDS-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148857#comment-17148857
]
Aravindan Vijayan commented on HDDS-3873:
-
[~adoroszlai] We may be ok to loosen the pipelines check in Recon. '_pipelines
> 0'_ is good enough since we have integration tests & unit tests covering the
same flow.
> Intermittent failure in Recon acceptance test due to too many pipelines
> ---
>
> Key: HDDS-3873
> URL: https://issues.apache.org/jira/browse/HDDS-3873
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
>
> Recon API acceptance test has too strict check on number of pipelines:
> {code}
> Check if Recon picks up DN heartbeats | FAIL |
> ...
> {"pipelines":5,...' does not contain '"pipelines":4'
> {code}
> https://github.com/apache/hadoop-ozone/pull/1050/checks?check_run_id=810139657
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3894) Noisy log at OM when the requested sequence is not written into DB
[
https://issues.apache.org/jira/browse/HDDS-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-3894:
Component/s: Ozone Recon
> Noisy log at OM when the requested sequence is not written into DB
> --
>
> Key: HDDS-3894
> URL: https://issues.apache.org/jira/browse/HDDS-3894
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager, Ozone Recon
>Reporter: Rakesh Radhakrishnan
>Assignee: Rakesh Radhakrishnan
>Priority: Minor
>
> Too many logs at OM, which is noisy. It looks like Recon is making this call
> to OM to update the delta info. OM can ignore this exception and reduce the
> log priority to DEBUG or so.
> *Ozone-om.log*
> {code:java}
> 2020-06-26 10:02:52,963 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolled log segment from
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_106536
> to
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_106536-106603
> 2020-06-26 10:02:52,982 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> created new log segment
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_106604
> 2020-06-26 10:09:54,828 ERROR org.apache.hadoop.hdds.utils.db.RDBStore:
> Unable to get delta updates since sequenceNumber 11268864
> org.rocksdb.RocksDBException: Requested sequence not yet written in the db
> at org.rocksdb.RocksDB.getUpdatesSince(Native Method)
> at org.rocksdb.RocksDB.getUpdatesSince(RocksDB.java:3588)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.getUpdatesSince(RDBStore.java:339)
> at
> org.apache.hadoop.ozone.om.OzoneManager.getDBUpdates(OzoneManager.java:3422)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getOMDBUpdates(OzoneManagerRequestHandler.java:257)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:194)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:171)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:109)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:74)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:99)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
> 2020-06-26 10:11:08,442 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolling segment log-106604_106637 to index:106637
> 2020-06-26 10:11:08,443 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolling segment log-106638_106654 to index:106654
> 2020-06-26 10:11:08,444 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolling segment log-106655_106671 to index:106671
> {code}
> *Recon.log*
> {code:java}
> 2020-06-26 10:08:55,883 INFO
> org.apache.hadoop.ozone.recon.fsck.MissingContainerTask: Missing Container
> task Thread took 88 milliseconds for processing 0 containers.
> 2020-06-26 10:09:54,824 INFO
> org.apache.hadoop.ozone.recon.spi.impl.OzoneManagerServiceProviderImpl:
> Syncing data from Ozone Manager.
> 2020-06-26 10:09:54,824 INFO
[jira] [Commented] (HDDS-3894) Noisy log at OM when the requested sequence is not written into DB
[
https://issues.apache.org/jira/browse/HDDS-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148855#comment-17148855
]
Aravindan Vijayan commented on HDDS-3894:
-
[~rakeshr] Thanks for tracking this issue. Since Recon can recover from this,
and the frequency of the getUpdatesSince call should be around ~10mins, we may
be ok with just suppressing the logs here. But we should understand why this is
happening.
Can you share more details on the setup here?
* Is OM HA enabled? Can we check if Recon was talking to the leader/follower OM?
* What is the load on OzoneManager when this happens? Are we running a stress
test on OM RocksDB?
> Noisy log at OM when the requested sequence is not written into DB
> --
>
> Key: HDDS-3894
> URL: https://issues.apache.org/jira/browse/HDDS-3894
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Reporter: Rakesh Radhakrishnan
>Assignee: Rakesh Radhakrishnan
>Priority: Minor
>
> Too many logs at OM, which is noisy. It looks like Recon is making this call
> to OM to update the delta info. OM can ignore this exception and reduce the
> log priority to DEBUG or so.
> *Ozone-om.log*
> {code:java}
> 2020-06-26 10:02:52,963 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolled log segment from
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_106536
> to
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_106536-106603
> 2020-06-26 10:02:52,982 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> created new log segment
> /data/3/jun27_new1/hadoop-ozone/om/ratis2/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_106604
> 2020-06-26 10:09:54,828 ERROR org.apache.hadoop.hdds.utils.db.RDBStore:
> Unable to get delta updates since sequenceNumber 11268864
> org.rocksdb.RocksDBException: Requested sequence not yet written in the db
> at org.rocksdb.RocksDB.getUpdatesSince(Native Method)
> at org.rocksdb.RocksDB.getUpdatesSince(RocksDB.java:3588)
> at
> org.apache.hadoop.hdds.utils.db.RDBStore.getUpdatesSince(RDBStore.java:339)
> at
> org.apache.hadoop.ozone.om.OzoneManager.getDBUpdates(OzoneManager.java:3422)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getOMDBUpdates(OzoneManagerRequestHandler.java:257)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:194)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:171)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:109)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:74)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:99)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
> 2020-06-26 10:11:08,442 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolling segment log-106604_106637 to index:106637
> 2020-06-26 10:11:08,443 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 5dca58f5-6231-4858-9589-4e64b1435aa4@group-C5BA1605619E-SegmentedRaftLogWorker:
> Rolling segment log-106638_106654 to index:106654
> 2020-06-26 10:11:08,444 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
>
[jira] [Commented] (HDDS-3402) Use proper acls for sub directories created during CreateDirectory operation
[ https://issues.apache.org/jira/browse/HDDS-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148843#comment-17148843 ] Rakesh Radhakrishnan commented on HDDS-3402: Thanks [~Sammi] for the interest. I am checking this. > Use proper acls for sub directories created during CreateDirectory operation > > > Key: HDDS-3402 > URL: https://issues.apache.org/jira/browse/HDDS-3402 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Rakesh Radhakrishnan >Priority: Blocker > Labels: TriagePending > > Use proper ACLS for subdirectories created during create directory operation. > All subdirectories/missing directories should inherit the ACLS from the > bucket if ancestors are not present in key table. If present should inherit > the ACLS from its ancestor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.
captainzmc commented on pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#issuecomment-651846590 Hi @xiaoyuyao，This PR implementation is consistent with HDDS-3286 batchDelete. And I split batchRename into two subtasks. This one, mainly OM side implementation. Could you help review it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1149: Hdds 3878
codecov-commenter commented on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-651845990 # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1149?src=pr=h1) Report > Merging [#1149](https://codecov.io/gh/apache/hadoop-ozone/pull/1149?src=pr=desc) into [master](https://codecov.io/gh/apache/hadoop-ozone/commit/f8fcc4760b227308c4a72f06f2e35864e69a1f22=desc) will **increase** coverage by `2.65%`. > The diff coverage is `69.44%`. [](https://codecov.io/gh/apache/hadoop-ozone/pull/1149?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#1149 +/- ## + Coverage 70.56% 73.21% +2.65% - Complexity 9427 9955 +528 Files 965 969 +4 Lines 4906349488 +425 Branches 4803 4865 +62 + Hits 3462036235+1615 + Misses1213710937-1200 - Partials 2306 2316 +10 ``` | [Impacted Files](https://codecov.io/gh/apache/hadoop-ozone/pull/1149?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../java/org/apache/hadoop/ozone/OzoneConfigKeys.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9oYWRvb3Avb3pvbmUvT3pvbmVDb25maWdLZXlzLmphdmE=) | `100.00% <ø> (ø)` | `1.00 <0.00> (ø)` | | | [...hadoop/hdds/scm/container/SCMContainerManager.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL2NvbnRhaW5lci9TQ01Db250YWluZXJNYW5hZ2VyLmphdmE=) | `72.54% <0.00%> (+1.96%)` | `35.00 <0.00> (+1.00)` | | | [...nt/algorithms/ContainerPlacementPolicyFactory.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLWhkZHMvc2VydmVyLXNjbS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL2hkZHMvc2NtL2NvbnRhaW5lci9wbGFjZW1lbnQvYWxnb3JpdGhtcy9Db250YWluZXJQbGFjZW1lbnRQb2xpY3lGYWN0b3J5LmphdmE=) | `83.33% <ø> (ø)` | `2.00 <0.00> (ø)` | | | [...src/main/java/org/apache/hadoop/ozone/OmUtils.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL09tVXRpbHMuamF2YQ==) | `76.98% <0.00%> (+1.77%)` | `40.00 <0.00> (+2.00)` | | | [.../java/org/apache/hadoop/ozone/om/OMConfigKeys.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL29tL09NQ29uZmlnS2V5cy5qYXZh) | `100.00% <ø> (ø)` | `1.00 <0.00> (ø)` | | | [...hadoop/ozone/om/protocol/OzoneManagerProtocol.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLW96b25lL2NvbW1vbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGFkb29wL296b25lL29tL3Byb3RvY29sL096b25lTWFuYWdlclByb3RvY29sLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (ø)` | | | [...g/apache/hadoop/ozone/om/codec/OMDBDefinition.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9jb2RlYy9PTURCRGVmaW5pdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...one/om/response/bucket/OMBucketCreateResponse.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9yZXNwb25zZS9idWNrZXQvT01CdWNrZXRDcmVhdGVSZXNwb25zZS5qYXZh) | `100.00% <ø> (ø)` | `4.00 <0.00> (ø)` | | | [...one/om/response/bucket/OMBucketDeleteResponse.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9yZXNwb25zZS9idWNrZXQvT01CdWNrZXREZWxldGVSZXNwb25zZS5qYXZh) | `100.00% <ø> (ø)` | `5.00 <0.00> (ø)` | | | [...m/response/bucket/OMBucketSetPropertyResponse.java](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree#diff-aGFkb29wLW96b25lL296b25lLW1hbmFnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2hhZG9vcC9vem9uZS9vbS9yZXNwb25zZS9idWNrZXQvT01CdWNrZXRTZXRQcm9wZXJ0eVJlc3BvbnNlLmphdmE=) | `100.00% <ø> (ø)` | `3.00 <0.00> (ø)` | | | ... and [302 more](https://codecov.io/gh/apache/hadoop-ozone/pull/1149/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1149?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) >
[jira] [Updated] (HDDS-3903) OzoneRpcClient support batch rename keys.
[ https://issues.apache.org/jira/browse/HDDS-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3903: - Labels: pull-request-available (was: ) > OzoneRpcClient support batch rename keys. > - > > Key: HDDS-3903 > URL: https://issues.apache.org/jira/browse/HDDS-3903 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: mingchao zhao >Assignee: mingchao zhao >Priority: Major > Labels: pull-request-available > > Currently rename folder is to get all the keys, and then rename them one by > one. This makes for poor performance. > HDDS-2939 can able to optimize this part, but at present the HDDS-2939 is > slow and still a long way to go. So we optimized the batch operation based on > the current interface. We were able to get better performance with this PR > before the HDDS-2939 came in. > This patch is a subtask of Batch Rename and first makes OzoneRpcClient > Support Batch Rename Keys. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.
captainzmc opened a new pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150 ## What changes were proposed in this pull request? Currently rename folder is to get all the keys, and then rename them one by one. This makes for poor performance. HDDS-2939 can able to optimize this part, but at present the HDDS-2939 is slow and still a long way to go. So we optimized the batch operation based on the current interface. We were able to get better performance with this PR before the HDDS-2939 came in. This PR is a subtask of Batch Rename and first makes OzoneRpcClient Support Batch Rename Keys. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3903 ## How was this patch tested? UT has been added. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3903) OzoneRpcClient support batch rename keys.
[ https://issues.apache.org/jira/browse/HDDS-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao updated HDDS-3903: Component/s: Ozone Manager > OzoneRpcClient support batch rename keys. > - > > Key: HDDS-3903 > URL: https://issues.apache.org/jira/browse/HDDS-3903 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: mingchao zhao >Assignee: mingchao zhao >Priority: Major > > Currently rename folder is to get all the keys, and then rename them one by > one. This makes for poor performance. > HDDS-2939 can able to optimize this part, but at present the HDDS-2939 is > slow and still a long way to go. So we optimized the batch operation based on > the current interface. We were able to get better performance with this PR > before the HDDS-2939 came in. > This patch is a subtask of Batch Rename and first makes OzoneRpcClient > Support Batch Rename Keys. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3904) BasicOzoneFileSystem support batchRename folder.
mingchao zhao created HDDS-3904: --- Summary: BasicOzoneFileSystem support batchRename folder. Key: HDDS-3904 URL: https://issues.apache.org/jira/browse/HDDS-3904 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Filesystem Reporter: mingchao zhao Currently rename folder is to get all the keys, and then rename them one by one. This makes for poor performance. HDDS-2939 can able to optimize this part, but at present the HDDS-2939 is slow and still a long way to go. So we optimized the batch operation based on the current interface. We were able to get better performance with this PR before the HDDS-2939 came in. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-3903) OzoneRpcClient support batch rename keys.
mingchao zhao created HDDS-3903: --- Summary: OzoneRpcClient support batch rename keys. Key: HDDS-3903 URL: https://issues.apache.org/jira/browse/HDDS-3903 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: mingchao zhao Assignee: mingchao zhao Currently rename folder is to get all the keys, and then rename them one by one. This makes for poor performance. HDDS-2939 can able to optimize this part, but at present the HDDS-2939 is slow and still a long way to go. So we optimized the batch operation based on the current interface. We were able to get better performance with this PR before the HDDS-2939 came in. This patch is a subtask of Batch Rename and first makes OzoneRpcClient Support Batch Rename Keys. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3902) OM HA client failover switcher to a wrong OM server
[
https://issues.apache.org/jira/browse/HDDS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-3902:
--
Labels: 0.7.0 (was: )
> OM HA client failover switcher to a wrong OM server
> ---
>
> Key: HDDS-3902
> URL: https://issues.apache.org/jira/browse/HDDS-3902
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: OM HA
>Reporter: Marton Elek
>Priority: Major
> Labels: 0.7.0
>
> Found this problem with the PR/branch HDDS-3878, but it seems to be
> independent.
> 1. ozone sh volume create /vol1 works well with HA
> 2. ozone freon omkg (rpc client) doesn't work
> {code}
> ozone freon omkg | grep "Failing over"
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 1, nodeId: om2
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:15:34 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 0, nodeId: omNodeIdDummy
> {code}
> om2 seems to be the leader but for some reason the failover logic switching
> back to an unknown node (?)
> {code}
> 2020-06-30 14:16:35 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:16:35 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:35 DEBUG Client:497 - The ping interval is 6 ms.
> 2020-06-30 14:16:35 DEBUG Client:795 - Connecting to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862
> 2020-06-30 14:16:35 DEBUG Client:1074 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root:
> starting, having connections 3
> 2020-06-30 14:16:35 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #0
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #0
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took
> 439ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #1
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #1
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 2ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #2 org.apache.hadoop.ozone.om.pro
> tocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #2
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #3
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #3
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:36 DEBUG Groups:312 - GroupCacheLoader - load.
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #5
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #11
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #8
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending
[jira] [Commented] (HDDS-3902) OM HA client failover switcher to a wrong OM server
[
https://issues.apache.org/jira/browse/HDDS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148727#comment-17148727
]
Marton Elek commented on HDDS-3902:
---
Seems to be a freon/validation problem. omServiceId is null, but there is no
validation in the OmClient. OzoneClient works well.
> OM HA client failover switcher to a wrong OM server
> ---
>
> Key: HDDS-3902
> URL: https://issues.apache.org/jira/browse/HDDS-3902
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: OM HA
>Reporter: Marton Elek
>Priority: Major
>
> Found this problem with the PR/branch HDDS-3878, but it seems to be
> independent.
> 1. ozone sh volume create /vol1 works well with HA
> 2. ozone freon omkg (rpc client) doesn't work
> {code}
> ozone freon omkg | grep "Failing over"
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 1, nodeId: om2
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:15:34 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 0, nodeId: omNodeIdDummy
> {code}
> om2 seems to be the leader but for some reason the failover logic switching
> back to an unknown node (?)
> {code}
> 2020-06-30 14:16:35 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:16:35 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:35 DEBUG Client:497 - The ping interval is 6 ms.
> 2020-06-30 14:16:35 DEBUG Client:795 - Connecting to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862
> 2020-06-30 14:16:35 DEBUG Client:1074 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root:
> starting, having connections 3
> 2020-06-30 14:16:35 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #0
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #0
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took
> 439ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #1
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #1
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 2ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #2 org.apache.hadoop.ozone.om.pro
> tocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #2
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #3
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #3
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:36 DEBUG Groups:312 - GroupCacheLoader - load.
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #5
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #11
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #8
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC
[jira] [Updated] (HDDS-3902) OM HA client failover switcher to a wrong OM server
[
https://issues.apache.org/jira/browse/HDDS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-3902:
--
Issue Type: Bug (was: Improvement)
> OM HA client failover switcher to a wrong OM server
> ---
>
> Key: HDDS-3902
> URL: https://issues.apache.org/jira/browse/HDDS-3902
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: OM HA
>Reporter: Marton Elek
>Priority: Blocker
> Labels: 0.7.0
>
> Found this problem with the PR/branch HDDS-3878, but it seems to be
> independent.
> 1. ozone sh volume create /vol1 works well with HA
> 2. ozone freon omkg (rpc client) doesn't work
> {code}
> ozone freon omkg | grep "Failing over"
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 1, nodeId: om2
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:15:34 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 0, nodeId: omNodeIdDummy
> {code}
> om2 seems to be the leader but for some reason the failover logic switching
> back to an unknown node (?)
> {code}
> 2020-06-30 14:16:35 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:16:35 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:35 DEBUG Client:497 - The ping interval is 6 ms.
> 2020-06-30 14:16:35 DEBUG Client:795 - Connecting to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862
> 2020-06-30 14:16:35 DEBUG Client:1074 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root:
> starting, having connections 3
> 2020-06-30 14:16:35 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #0
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #0
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took
> 439ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #1
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #1
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 2ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #2 org.apache.hadoop.ozone.om.pro
> tocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #2
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #3
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #3
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:36 DEBUG Groups:312 - GroupCacheLoader - load.
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #5
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #11
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #8
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
>
[jira] [Updated] (HDDS-3902) OM HA client failover switcher to a wrong OM server
[
https://issues.apache.org/jira/browse/HDDS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-3902:
--
Priority: Blocker (was: Major)
> OM HA client failover switcher to a wrong OM server
> ---
>
> Key: HDDS-3902
> URL: https://issues.apache.org/jira/browse/HDDS-3902
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: OM HA
>Reporter: Marton Elek
>Priority: Blocker
> Labels: 0.7.0
>
> Found this problem with the PR/branch HDDS-3878, but it seems to be
> independent.
> 1. ozone sh volume create /vol1 works well with HA
> 2. ozone freon omkg (rpc client) doesn't work
> {code}
> ozone freon omkg | grep "Failing over"
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 1, nodeId: om2
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:15:34 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 0, nodeId: omNodeIdDummy
> {code}
> om2 seems to be the leader but for some reason the failover logic switching
> back to an unknown node (?)
> {code}
> 2020-06-30 14:16:35 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:16:35 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:35 DEBUG Client:497 - The ping interval is 6 ms.
> 2020-06-30 14:16:35 DEBUG Client:795 - Connecting to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862
> 2020-06-30 14:16:35 DEBUG Client:1074 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root:
> starting, having connections 3
> 2020-06-30 14:16:35 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #0
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #0
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took
> 439ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #1
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #1
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 2ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #2 org.apache.hadoop.ozone.om.pro
> tocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #2
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #3
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #3
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:36 DEBUG Groups:312 - GroupCacheLoader - load.
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #5
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #11
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #8
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
[jira] [Updated] (HDDS-3902) OM HA client failover switcher to a wrong OM server
[
https://issues.apache.org/jira/browse/HDDS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-3902:
--
Target Version/s: (was: 0.6.0)
> OM HA client failover switcher to a wrong OM server
> ---
>
> Key: HDDS-3902
> URL: https://issues.apache.org/jira/browse/HDDS-3902
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: OM HA
>Reporter: Marton Elek
>Priority: Major
>
> Found this problem with the PR/branch HDDS-3878, but it seems to be
> independent.
> 1. ozone sh volume create /vol1 works well with HA
> 2. ozone freon omkg (rpc client) doesn't work
> {code}
> ozone freon omkg | grep "Failing over"
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 1, nodeId: om2
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:15:34 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 0, nodeId: omNodeIdDummy
> {code}
> om2 seems to be the leader but for some reason the failover logic switching
> back to an unknown node (?)
> {code}
> 2020-06-30 14:16:35 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:16:35 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:35 DEBUG Client:497 - The ping interval is 6 ms.
> 2020-06-30 14:16:35 DEBUG Client:795 - Connecting to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862
> 2020-06-30 14:16:35 DEBUG Client:1074 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root:
> starting, having connections 3
> 2020-06-30 14:16:35 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #0
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #0
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took
> 439ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #1
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #1
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 2ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #2 org.apache.hadoop.ozone.om.pro
> tocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #2
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #3
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #3
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:36 DEBUG Groups:312 - GroupCacheLoader - load.
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #5
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #11
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #8
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #12
>
[jira] [Updated] (HDDS-3902) OM HA client failover switcher to a wrong OM server
[
https://issues.apache.org/jira/browse/HDDS-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-3902:
--
Priority: Major (was: Blocker)
> OM HA client failover switcher to a wrong OM server
> ---
>
> Key: HDDS-3902
> URL: https://issues.apache.org/jira/browse/HDDS-3902
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: OM HA
>Reporter: Marton Elek
>Priority: Major
>
> Found this problem with the PR/branch HDDS-3878, but it seems to be
> independent.
> 1. ozone sh volume create /vol1 works well with HA
> 2. ozone freon omkg (rpc client) doesn't work
> {code}
> ozone freon omkg | grep "Failing over"
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 1, nodeId: om2
> 2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:15:34 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 0, nodeId: omNodeIdDummy
> {code}
> om2 seems to be the leader but for some reason the failover logic switching
> back to an unknown node (?)
> {code}
> 2020-06-30 14:16:35 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
> to index: 2, nodeId: om3
> 2020-06-30 14:16:35 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:35 DEBUG Client:497 - The ping interval is 6 ms.
> 2020-06-30 14:16:35 DEBUG Client:795 - Connecting to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862
> 2020-06-30 14:16:35 DEBUG Client:1074 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root:
> starting, having connections 3
> 2020-06-30 14:16:35 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #0
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #0
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took
> 439ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #1
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #1
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 2ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #2 org.apache.hadoop.ozone.om.pro
> tocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #2
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
> sending #3
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
> ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
> value #3
> 2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
> 2020-06-30 14:16:36 DEBUG Client:63 - getting client out of cache:
> org.apache.hadoop.ipc.Client@f5acb9d
> 2020-06-30 14:16:36 DEBUG Groups:312 - GroupCacheLoader - load.
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #5
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #11
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #8
> org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
> 2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
> ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
> sending #12
>
[jira] [Created] (HDDS-3902) OM HA client failover switcher to a wrong OM server
Marton Elek created HDDS-3902:
-
Summary: OM HA client failover switcher to a wrong OM server
Key: HDDS-3902
URL: https://issues.apache.org/jira/browse/HDDS-3902
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: OM HA
Reporter: Marton Elek
Found this problem with the PR/branch HDDS-3878, but it seems to be independent.
1. ozone sh volume create /vol1 works well with HA
2. ozone freon omkg (rpc client) doesn't work
{code}
ozone freon omkg | grep "Failing over"
2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
to index: 1, nodeId: om2
2020-06-30 14:15:31 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
to index: 2, nodeId: om3
2020-06-30 14:15:34 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
to index: 0, nodeId: omNodeIdDummy
{code}
om2 seems to be the leader but for some reason the failover logic switching
back to an unknown node (?)
{code}
2020-06-30 14:16:35 DEBUG OMFailoverProxyProvider:271 - Failing over OM proxy
to index: 2, nodeId: om3
2020-06-30 14:16:35 DEBUG Client:63 - getting client out of cache:
org.apache.hadoop.ipc.Client@f5acb9d
2020-06-30 14:16:35 DEBUG Client:497 - The ping interval is 6 ms.
2020-06-30 14:16:35 DEBUG Client:795 - Connecting to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862
2020-06-30 14:16:35 DEBUG Client:1074 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root:
starting, having connections 3
2020-06-30 14:16:35 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
sending #0
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
value #0
2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 439ms
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
sending #1
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
value #1
2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 2ms
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
sending #2 org.apache.hadoop.ozone.om.pro
tocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
value #2
2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root
sending #3
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1191 - IPC Client (363509958) connection to
ozone-om-2.ozone-om.default.svc.cluster.local/10.42.0.175:9862 from root got
value #3
2020-06-30 14:16:36 DEBUG ProtobufRpcEngine:254 - Call: submitRequest took 1ms
2020-06-30 14:16:36 DEBUG Client:63 - getting client out of cache:
org.apache.hadoop.ipc.Client@f5acb9d
2020-06-30 14:16:36 DEBUG Groups:312 - GroupCacheLoader - load.
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
sending #5
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
sending #11
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
sending #8
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
sending #12
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
ozone-om-0.ozone-om.default.svc.cluster.local/10.42.0.173:9862 from root
sending #10
org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2020-06-30 14:16:36 DEBUG Client:1137 - IPC Client (363509958) connection to
[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1053: HDDS-3737. Avoid serialization between UUID and String
runzhiwang commented on a change in pull request #1053:
URL: https://github.com/apache/hadoop-ozone/pull/1053#discussion_r447697031
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java
##
@@ -301,14 +306,21 @@ public static Pipeline
getFromProtobuf(HddsProtos.Pipeline pipeline)
for (DatanodeDetailsProto member : pipeline.getMembersList()) {
nodes.add(DatanodeDetails.getFromProtoBuf(member));
}
+UUID leaderId = null;
+if (pipeline.hasLeaderID() &&
+StringUtils.isNotEmpty(pipeline.getLeaderID())) {
+ leaderId = UUID.fromString(pipeline.getLeaderID());
Review comment:
@xiaoyuyao Thanks for review. I have updated the patch, Could you help
review it again ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on pull request #1053: HDDS-3737. Avoid serialization between UUID and String
runzhiwang commented on pull request #1053: URL: https://github.com/apache/hadoop-ozone/pull/1053#issuecomment-651802478 /ready This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2978) Intermittent failure in TestResourceLimitCache
[
https://issues.apache.org/jira/browse/HDDS-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2978:
-
Labels: pull-request-available (was: )
> Intermittent failure in TestResourceLimitCache
> --
>
> Key: HDDS-2978
> URL: https://issues.apache.org/jira/browse/HDDS-2978
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Neo Yang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> {code:title=https://github.com/adoroszlai/hadoop-ozone/runs/425954260}
> 2020-02-04T18:13:01.6043382Z [ERROR] Tests run: 1, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 0.393 s <<< FAILURE! - in
> org.apache.hadoop.hdds.utils.TestResourceLimitCache
> 2020-02-04T18:13:01.6044180Z [ERROR]
> testResourceLimitCache(org.apache.hadoop.hdds.utils.TestResourceLimitCache)
> Time elapsed: 0.285 s <<< FAILURE!
> 2020-02-04T18:13:01.6045063Z java.lang.AssertionError: expected null, but
> was:
> ...
> 2020-02-04T18:13:01.6072118Z at
> org.apache.hadoop.hdds.utils.TestResourceLimitCache.testResourceLimitCache(TestResourceLimitCache.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] cku328 commented on pull request #1146: HDDS-2978. Intermittent failure in TestResourceLimitCache
cku328 commented on pull request #1146: URL: https://github.com/apache/hadoop-ozone/pull/1146#issuecomment-651774937 Thanks @adoroszlai @lokeshj1703 for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek opened a new pull request #1149: Hdds 3878
elek opened a new pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149 ## What changes were proposed in this pull request? _emphasized text_om.serviceId is required on case of OM.HA in all the client parameters even if there is only one om.serviceId and it can be chosen. My goal is: 1. Provide better usability 2. Simplify the documentation task ;-) With using the om.serviceId from the config if 1. config is available 2. om ha is configured 3. only one service is configured It also makes easier to run the same tests with/without HA ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3878?filter=12349091 ## How was this patch tested? 1. Got full green build on my fork. 2. ozone-ha acceptance tests ozone-ha acceptance tests are turned off long time ago (it included some life-cycle test to start/stop services. In this patch I simplified the `ha` cluster and added simple smoketest. Later we can restore the lifecycle tests (start/stop) but I would prefer to use a generic approach for all the clusters. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-2978) Intermittent failure in TestResourceLimitCache
[
https://issues.apache.org/jira/browse/HDDS-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai resolved HDDS-2978.
Fix Version/s: 0.6.0
Resolution: Fixed
> Intermittent failure in TestResourceLimitCache
> --
>
> Key: HDDS-2978
> URL: https://issues.apache.org/jira/browse/HDDS-2978
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Neo Yang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> {code:title=https://github.com/adoroszlai/hadoop-ozone/runs/425954260}
> 2020-02-04T18:13:01.6043382Z [ERROR] Tests run: 1, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 0.393 s <<< FAILURE! - in
> org.apache.hadoop.hdds.utils.TestResourceLimitCache
> 2020-02-04T18:13:01.6044180Z [ERROR]
> testResourceLimitCache(org.apache.hadoop.hdds.utils.TestResourceLimitCache)
> Time elapsed: 0.285 s <<< FAILURE!
> 2020-02-04T18:13:01.6045063Z java.lang.AssertionError: expected null, but
> was:
> ...
> 2020-02-04T18:13:01.6072118Z at
> org.apache.hadoop.hdds.utils.TestResourceLimitCache.testResourceLimitCache(TestResourceLimitCache.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2978) Intermittent failure in TestResourceLimitCache
[
https://issues.apache.org/jira/browse/HDDS-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-2978:
---
Labels: (was: pull-request-available)
> Intermittent failure in TestResourceLimitCache
> --
>
> Key: HDDS-2978
> URL: https://issues.apache.org/jira/browse/HDDS-2978
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Neo Yang
>Priority: Minor
>
> {code:title=https://github.com/adoroszlai/hadoop-ozone/runs/425954260}
> 2020-02-04T18:13:01.6043382Z [ERROR] Tests run: 1, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 0.393 s <<< FAILURE! - in
> org.apache.hadoop.hdds.utils.TestResourceLimitCache
> 2020-02-04T18:13:01.6044180Z [ERROR]
> testResourceLimitCache(org.apache.hadoop.hdds.utils.TestResourceLimitCache)
> Time elapsed: 0.285 s <<< FAILURE!
> 2020-02-04T18:13:01.6045063Z java.lang.AssertionError: expected null, but
> was:
> ...
> 2020-02-04T18:13:01.6072118Z at
> org.apache.hadoop.hdds.utils.TestResourceLimitCache.testResourceLimitCache(TestResourceLimitCache.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2978) Intermittent failure in TestResourceLimitCache
[
https://issues.apache.org/jira/browse/HDDS-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2978:
-
Labels: pull-request-available (was: )
> Intermittent failure in TestResourceLimitCache
> --
>
> Key: HDDS-2978
> URL: https://issues.apache.org/jira/browse/HDDS-2978
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Neo Yang
>Priority: Minor
> Labels: pull-request-available
>
> {code:title=https://github.com/adoroszlai/hadoop-ozone/runs/425954260}
> 2020-02-04T18:13:01.6043382Z [ERROR] Tests run: 1, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 0.393 s <<< FAILURE! - in
> org.apache.hadoop.hdds.utils.TestResourceLimitCache
> 2020-02-04T18:13:01.6044180Z [ERROR]
> testResourceLimitCache(org.apache.hadoop.hdds.utils.TestResourceLimitCache)
> Time elapsed: 0.285 s <<< FAILURE!
> 2020-02-04T18:13:01.6045063Z java.lang.AssertionError: expected null, but
> was:
> ...
> 2020-02-04T18:13:01.6072118Z at
> org.apache.hadoop.hdds.utils.TestResourceLimitCache.testResourceLimitCache(TestResourceLimitCache.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2978) Intermittent failure in TestResourceLimitCache
[
https://issues.apache.org/jira/browse/HDDS-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-2978:
---
Labels: (was: pull-request-available)
> Intermittent failure in TestResourceLimitCache
> --
>
> Key: HDDS-2978
> URL: https://issues.apache.org/jira/browse/HDDS-2978
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Neo Yang
>Priority: Minor
> Fix For: 0.6.0
>
>
> {code:title=https://github.com/adoroszlai/hadoop-ozone/runs/425954260}
> 2020-02-04T18:13:01.6043382Z [ERROR] Tests run: 1, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 0.393 s <<< FAILURE! - in
> org.apache.hadoop.hdds.utils.TestResourceLimitCache
> 2020-02-04T18:13:01.6044180Z [ERROR]
> testResourceLimitCache(org.apache.hadoop.hdds.utils.TestResourceLimitCache)
> Time elapsed: 0.285 s <<< FAILURE!
> 2020-02-04T18:13:01.6045063Z java.lang.AssertionError: expected null, but
> was:
> ...
> 2020-02-04T18:13:01.6072118Z at
> org.apache.hadoop.hdds.utils.TestResourceLimitCache.testResourceLimitCache(TestResourceLimitCache.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2978) Intermittent failure in TestResourceLimitCache
[
https://issues.apache.org/jira/browse/HDDS-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-2978:
---
Target Version/s: (was: 0.7.0)
> Intermittent failure in TestResourceLimitCache
> --
>
> Key: HDDS-2978
> URL: https://issues.apache.org/jira/browse/HDDS-2978
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Neo Yang
>Priority: Minor
> Fix For: 0.6.0
>
>
> {code:title=https://github.com/adoroszlai/hadoop-ozone/runs/425954260}
> 2020-02-04T18:13:01.6043382Z [ERROR] Tests run: 1, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 0.393 s <<< FAILURE! - in
> org.apache.hadoop.hdds.utils.TestResourceLimitCache
> 2020-02-04T18:13:01.6044180Z [ERROR]
> testResourceLimitCache(org.apache.hadoop.hdds.utils.TestResourceLimitCache)
> Time elapsed: 0.285 s <<< FAILURE!
> 2020-02-04T18:13:01.6045063Z java.lang.AssertionError: expected null, but
> was:
> ...
> 2020-02-04T18:13:01.6072118Z at
> org.apache.hadoop.hdds.utils.TestResourceLimitCache.testResourceLimitCache(TestResourceLimitCache.java:85)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai merged pull request #1146: HDDS-2978. Intermittent failure in TestResourceLimitCache
adoroszlai merged pull request #1146: URL: https://github.com/apache/hadoop-ozone/pull/1146 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1146: HDDS-2978. Intermittent failure in TestResourceLimitCache
adoroszlai commented on pull request #1146: URL: https://github.com/apache/hadoop-ozone/pull/1146#issuecomment-651746203 Thanks @cku328 for the fix and @lokeshj1703 for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1139: HDDS-3885. Create Datanode home page
adoroszlai commented on pull request #1139: URL: https://github.com/apache/hadoop-ozone/pull/1139#issuecomment-651743270 Thanks @maobaolong for updating the patch. Let's wait for others to take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1139: HDDS-3885. Create Datanode home page
maobaolong commented on pull request #1139: URL: https://github.com/apache/hadoop-ozone/pull/1139#issuecomment-651742566 @adoroszlai Thanks for your review. AFAIK, the reason why Rpc Metrics page is empty for datanode is that datanode has no rpc service.PTAL. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1128: HDDS-3862. Prepare checks for running some tests multiple times
adoroszlai commented on a change in pull request #1128:
URL: https://github.com/apache/hadoop-ozone/pull/1128#discussion_r447618921
##
File path: hadoop-ozone/dev-support/checks/junit.sh
##
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -u -o pipefail
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+cd "$DIR/../../.." || exit 1
+
+: ${CHECK:="unit"}
+: ${ITERATIONS:="1"}
+
+export MAVEN_OPTS="-Xmx4096m"
+MAVEN_OPTIONS='-B -Dskip.npx -Dskip.installnpx'
+mvn ${MAVEN_OPTIONS} -DskipTests clean install
+
+REPORT_DIR=${OUTPUT_DIR:-"$DIR/../../../target/${CHECK}"}
+mkdir -p "$REPORT_DIR"
+
+rc=0
+for i in $(seq 1 ${ITERATIONS}); do
+ if [[ ${ITERATIONS} -gt 1 ]]; then
+original_report_dir="${REPORT_DIR}"
+REPORT_DIR="${original_report_dir}/iteration${i}"
+mkdir -p "${REPORT_DIR}"
+ fi
+
+ mvn ${MAVEN_OPTIONS} -fae "$@" test \
+| tee "${REPORT_DIR}/output.log"
+ irc=$?
+
+ # shellcheck source=hadoop-ozone/dev-support/checks/_mvn_unit_report.sh
+ source "${DIR}/_mvn_unit_report.sh"
+
+ if [[ ${ITERATIONS} -gt 1 ]]; then
+REPORT_DIR="${original_report_dir}"
+echo "Iteration ${i} exit code: ${irc}" | tee -a "${REPORT_DIR}/output.log"
Review comment:
I've updated the patch based on this suggestion. Now that I see the
[results of a repeated
run](https://github.com/adoroszlai/hadoop-ozone/runs/822046544), a question
occurred to me: should we print only failed iterations (to match how we only
list failed tests in summary) or all of them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1128: HDDS-3862. Prepare checks for running some tests multiple times
adoroszlai commented on a change in pull request #1128:
URL: https://github.com/apache/hadoop-ozone/pull/1128#discussion_r447618921
##
File path: hadoop-ozone/dev-support/checks/junit.sh
##
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -u -o pipefail
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+cd "$DIR/../../.." || exit 1
+
+: ${CHECK:="unit"}
+: ${ITERATIONS:="1"}
+
+export MAVEN_OPTS="-Xmx4096m"
+MAVEN_OPTIONS='-B -Dskip.npx -Dskip.installnpx'
+mvn ${MAVEN_OPTIONS} -DskipTests clean install
+
+REPORT_DIR=${OUTPUT_DIR:-"$DIR/../../../target/${CHECK}"}
+mkdir -p "$REPORT_DIR"
+
+rc=0
+for i in $(seq 1 ${ITERATIONS}); do
+ if [[ ${ITERATIONS} -gt 1 ]]; then
+original_report_dir="${REPORT_DIR}"
+REPORT_DIR="${original_report_dir}/iteration${i}"
+mkdir -p "${REPORT_DIR}"
+ fi
+
+ mvn ${MAVEN_OPTIONS} -fae "$@" test \
+| tee "${REPORT_DIR}/output.log"
+ irc=$?
+
+ # shellcheck source=hadoop-ozone/dev-support/checks/_mvn_unit_report.sh
+ source "${DIR}/_mvn_unit_report.sh"
+
+ if [[ ${ITERATIONS} -gt 1 ]]; then
+REPORT_DIR="${original_report_dir}"
+echo "Iteration ${i} exit code: ${irc}" | tee -a "${REPORT_DIR}/output.log"
Review comment:
I've updated the patch based on this suggestion. Now that I see the
results of a repeated run, a question occurred to me: should we print only
failed iterations (to match how we only list failed tests in summary) or all of
them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1133: HDDS-3875. Package classpath files to the jar files instead of uploading them as artifacts
elek commented on a change in pull request #1133:
URL: https://github.com/apache/hadoop-ozone/pull/1133#discussion_r447600314
##
File path: hadoop-ozone/dist/pom.xml
##
@@ -41,103 +41,13 @@
copy-classpath-files
prepare-package
- copy
+ unpack-dependencies
target/ozone-${ozone.version}/share/ozone/classpath
-
-
- org.apache.hadoop
- hadoop-hdds-server-scm
- ${hdds.version}
- classpath
- cp
- hadoop-hdds-server-scm.classpath
-
-
- org.apache.hadoop
- hadoop-hdds-tools
- ${hdds.version}
- classpath
- cp
- hadoop-hdds-tools.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-s3gateway
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-s3gateway.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-csi
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-csi.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-ozone-manager
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-ozone-manager.classpath
-
-
-
- org.apache.hadoop
- hadoop-ozone-tools
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-tools.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-filesystem
- ${ozone.version}
- classpath
- cp
-
hadoop-ozone-filesystem.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-common
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-common.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-datanode
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-datanode.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-upgrade
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-upgrade.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-insight
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-insight.classpath
-
-
+ *.classpath
Review comment:
Nice catch. I did a `grep` for `OZONE_RUN_ARTIFACT_NAME` in `ozone`
shell script and selected only those artifacts.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1115: HDDS-3632. starter scripts can't manage Ozone and HDFS datandodes on the same machine
elek commented on a change in pull request #1115:
URL: https://github.com/apache/hadoop-ozone/pull/1115#discussion_r447582657
##
File path: hadoop-ozone/dist/src/shell/hdds/hadoop-functions.sh
##
@@ -2702,11 +2702,11 @@ function hadoop_generic_java_subcmd_handler
priv_outfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
priv_errfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.err"
priv_pidfile="${HADOOP_PID_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid"
-
daemon_outfile="${HADOOP_LOG_DIR}/hadoop-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
-
daemon_pidfile="${HADOOP_PID_DIR}/hadoop-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid"
+
daemon_outfile="${HADOOP_LOG_DIR}/ozone-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
+
daemon_pidfile="${HADOOP_PID_DIR}/ozone-${HADOOP_SECURE_USER}-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid"
else
-
daemon_outfile="${HADOOP_LOG_DIR}/hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
-
daemon_pidfile="${HADOOP_PID_DIR}/hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid"
+
daemon_outfile="${HADOOP_LOG_DIR}/ozone-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
+
daemon_pidfile="${HADOOP_PID_DIR}/ozone-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid"
fi
# are we actually in daemon mode?
Review comment:
Hard to say. I guess it's not a common setup, but who knows, and it
should be safer to change, anyway.
Fixed, thanks the suggestion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1115: HDDS-3632. starter scripts can't manage Ozone and HDFS datandodes on the same machine
elek commented on a change in pull request #1115:
URL: https://github.com/apache/hadoop-ozone/pull/1115#discussion_r447581743
##
File path: hadoop-ozone/dist/src/shell/hdds/hadoop-functions.sh
##
@@ -2702,11 +2702,11 @@ function hadoop_generic_java_subcmd_handler
priv_outfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.out"
priv_errfile="${HADOOP_LOG_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}-${HOSTNAME}.err"
priv_pidfile="${HADOOP_PID_DIR}/privileged-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid"
Review comment:
Good point, thanks. Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1131: HDDS-3876. Display summary of failures as a separate job step
adoroszlai commented on pull request #1131: URL: https://github.com/apache/hadoop-ozone/pull/1131#issuecomment-651707810 Thanks @elek for reviewing and committing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 edited a comment on pull request #1121: HDDS-3432. Enable TestBlockDeletion test cases.
lokeshj1703 edited a comment on pull request #1121: URL: https://github.com/apache/hadoop-ozone/pull/1121#issuecomment-651706881 @adoroszlai The workflow is now green. There were 2 failures in around 70 runs. One where cluster start timed out at waitForClusterToBeReady and another with appendEntriesTimeout. https://github.com/lokeshj1703/hadoop-ozone/actions?query=workflow%3ATestBlockDeletion2 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1121: HDDS-3432. Enable TestBlockDeletion test cases.
lokeshj1703 commented on pull request #1121: URL: https://github.com/apache/hadoop-ozone/pull/1121#issuecomment-651706881 @adoroszlai The workflow is now green. There were 2 failures in around 70 runs. One where cluster start timed out at waitForClusterToBeReady and another with appendEntriesTimeout. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3246) Include OM hostname info in "getserviceroles" subcommand of OM CLI
[ https://issues.apache.org/jira/browse/HDDS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3246. --- Fix Version/s: 0.6.0 Resolution: Fixed > Include OM hostname info in "getserviceroles" subcommand of OM CLI > -- > > Key: HDDS-3246 > URL: https://issues.apache.org/jira/browse/HDDS-3246 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone CLI, Ozone Manager >Reporter: Nilotpal Nandi >Assignee: Nilotpal Nandi >Priority: Major > Labels: Triaged, pull-request-available > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, "getserviceroles" subcommand of OM CLI displays only node ID > along with its serviceRole. > ozone admin om getserviceroles -id=ozone1 > om2 : FOLLOWER > om3 : FOLLOWER > om1 : LEADER > Need to include Hostname info -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3876) Display summary of failures as a separate job step
[
https://issues.apache.org/jira/browse/HDDS-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-3876:
--
Fix Version/s: 0.6.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Display summary of failures as a separate job step
> --
>
> Key: HDDS-3876
> URL: https://issues.apache.org/jira/browse/HDDS-3876
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: CI
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
> Fix For: 0.6.0
>
>
> Most CI checks print a summary (failed tests, checkstyle/rat violations,
> etc.) to stdout at the end of the test run, as well as into {{summary.txt}}.
> Currently we have the following ways to view this output:
> * drill down to the test step, scroll past lots of output
> * download raw log, scroll past lots of output
> * download artifact, unzip, open {{summary.txt}}
> I propose displaying contents of {{summary.txt}} as a separate step.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #706: HDDS-3246. Include OM hostname info in getserviceroles subcommand of OM CLI
elek commented on pull request #706: URL: https://github.com/apache/hadoop-ozone/pull/706#issuecomment-651705520 88347c5 is also green. I am merging it now. Thanks @nilotpalnandi the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #706: HDDS-3246. Include OM hostname info in getserviceroles subcommand of OM CLI
elek merged pull request #706: URL: https://github.com/apache/hadoop-ozone/pull/706 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1131: HDDS-3876. Display summary of failures as a separate job step
elek merged pull request #1131: URL: https://github.com/apache/hadoop-ozone/pull/1131 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1131: HDDS-3876. Display summary of failures as a separate job step
elek commented on pull request #1131: URL: https://github.com/apache/hadoop-ozone/pull/1131#issuecomment-651704654 > We no longer have separate workflow for push and PR, so this change applies to PRs, too. Ups, thanks the info. I missed that PR, but like it (if it's not too late ;-) ) In this case, let me merge this PR now. Thanks the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang edited a comment on pull request #1028: HDDS-3735. Improve SCM performance with 3.7% by remove unnecessary lock and unlock
runzhiwang edited a comment on pull request #1028: URL: https://github.com/apache/hadoop-ozone/pull/1028#issuecomment-651703240 > Without write lock the container might be removed from ownerMap and not from the typeMap, which can cause inconsistency. @elek I think this case can not be avoided by only add read lock. Unless we read containerMap, ownerMap, typeMap in the single method with a read lock, Otherwise for example 1. we read containerMap, 2. we remove container, 3. we read ownerMap. It will also cause inconsistency, because container exists in step1 but not exists in step3. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on pull request #1028: HDDS-3735. Improve SCM performance with 3.7% by remove unnecessary lock and unlock
runzhiwang commented on pull request #1028: URL: https://github.com/apache/hadoop-ozone/pull/1028#issuecomment-651703240 > Without write lock the container might be removed from ownerMap and not from the typeMap, which can cause inconsistency. I think this case can not be avoided by only add read lock. Unless we read containerMap, ownerMap, typeMap in the single method with a read lock, Otherwise for example 1. we read containerMap, 2. we remove container, 3. we read ownerMap. It will also cause inconsistency, because container exists in step1 but not exists in step3. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1127: HDDS-3421. Enabling TestStorageContainerManager.
elek commented on pull request #1127: URL: https://github.com/apache/hadoop-ozone/pull/1127#issuecomment-651701000 > CI check is failing for "TestBlockOutputStreamWithFailuresFlushDelay" which is an unrelated failure. Please take a look. I agree, but I prefer to merge only green builds. If something is failing we should feel the pain and fix it. Triggered a new build. Build failures are build failures. Related to the PR or not. Its all the same. If I have to merge a PR, then I prefer to merge only green builds. (And fix ALL the intermittent failures on master ASAP).https://t.co/rMGXSSGcOL;>https://t.co/rMGXSSGcOL (@anzix) https://twitter.com/anzix/status/1246055720220573696?ref_src=twsrc%5Etfw;>April 3, 2020 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1127: HDDS-3421. Enabling TestStorageContainerManager.
elek commented on pull request #1127: URL: https://github.com/apache/hadoop-ozone/pull/1127#issuecomment-651699418 +1 Thanks the fix. If it's passed 20/20 the problem can be the too short timeout. Let's merge it and monitor the state in the next master builds. "Sors bona, nihil aliud" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3161) we need to block illegal characters when creating keys
[ https://issues.apache.org/jira/browse/HDDS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3161. --- Fix Version/s: 0.6.0 Resolution: Fixed > we need to block illegal characters when creating keys > -- > > Key: HDDS-3161 > URL: https://issues.apache.org/jira/browse/HDDS-3161 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Reporter: mingchao zhao >Assignee: Neo Yang >Priority: Major > Labels: Triaged, pull-request-available > Fix For: 0.6.0 > > Attachments: image-2020-03-11-20-35-16-762.png > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, ozone does not impose effective restrictions on illegal characters > when creating keys. > !image-2020-03-11-20-35-16-762.png|width=563,height=80! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #812: HDDS-3161. Block illegal characters when creating keys.
elek commented on pull request #812: URL: https://github.com/apache/hadoop-ozone/pull/812#issuecomment-651693879 :musical_note: Green build, green build, tada-tada-tada... :musical_note: Merging it right now, tada-tada-tada :musical_note: Thanks the patch @cku328 (and sorry for the long review time) and the review @adoroszlai This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #812: HDDS-3161. Block illegal characters when creating keys.
elek merged pull request #812: URL: https://github.com/apache/hadoop-ozone/pull/812 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3782) Remove podAntiAffinity from datanode-statefulset
[ https://issues.apache.org/jira/browse/HDDS-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3782. --- Fix Version/s: 0.6.0 Resolution: Fixed > Remove podAntiAffinity from datanode-statefulset > - > > Key: HDDS-3782 > URL: https://issues.apache.org/jira/browse/HDDS-3782 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: kubernetes >Affects Versions: 0.7.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > > Now, i think the yaml files under examples folder are trial purpose, so we > should not add the strict limitation to prevent people easy to taste the > example, if who want reference this example and use it in production env, > then they will do a lot of modification. > For this ticket, it suppose to remove the podAntiAffinity check to let on > node kubernetes run the ozone cluster on kubernetes successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1057: HDDS-3782. Remove podAntiAffinity from datanode-statefulset
elek merged pull request #1057: URL: https://github.com/apache/hadoop-ozone/pull/1057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1128: HDDS-3862. Prepare checks for running some tests multiple times
elek commented on a change in pull request #1128:
URL: https://github.com/apache/hadoop-ozone/pull/1128#discussion_r447558463
##
File path: hadoop-ozone/dev-support/checks/junit.sh
##
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -u -o pipefail
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
+cd "$DIR/../../.." || exit 1
+
+: ${CHECK:="unit"}
+: ${ITERATIONS:="1"}
+
+export MAVEN_OPTS="-Xmx4096m"
+MAVEN_OPTIONS='-B -Dskip.npx -Dskip.installnpx'
+mvn ${MAVEN_OPTIONS} -DskipTests clean install
+
+REPORT_DIR=${OUTPUT_DIR:-"$DIR/../../../target/${CHECK}"}
+mkdir -p "$REPORT_DIR"
+
+rc=0
+for i in $(seq 1 ${ITERATIONS}); do
+ if [[ ${ITERATIONS} -gt 1 ]]; then
+original_report_dir="${REPORT_DIR}"
+REPORT_DIR="${original_report_dir}/iteration${i}"
+mkdir -p "${REPORT_DIR}"
+ fi
+
+ mvn ${MAVEN_OPTIONS} -fae "$@" test \
+| tee "${REPORT_DIR}/output.log"
+ irc=$?
+
+ # shellcheck source=hadoop-ozone/dev-support/checks/_mvn_unit_report.sh
+ source "${DIR}/_mvn_unit_report.sh"
+
+ if [[ ${ITERATIONS} -gt 1 ]]; then
+REPORT_DIR="${original_report_dir}"
+echo "Iteration ${i} exit code: ${irc}" | tee -a "${REPORT_DIR}/output.log"
Review comment:
```suggestion
echo "Iteration ${i} exit code: ${irc}" | tee -a
"${REPORT_DIR}/summary.txt"
```
Based on the existing convention, output.log contains the full std output,
summary.txt the information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1133: HDDS-3875. Package classpath files to the jar files instead of uploading them as artifacts
adoroszlai commented on a change in pull request #1133:
URL: https://github.com/apache/hadoop-ozone/pull/1133#discussion_r447559058
##
File path: hadoop-ozone/dist/pom.xml
##
@@ -41,103 +41,13 @@
copy-classpath-files
prepare-package
- copy
+ unpack-dependencies
target/ozone-${ozone.version}/share/ozone/classpath
-
-
- org.apache.hadoop
- hadoop-hdds-server-scm
- ${hdds.version}
- classpath
- cp
- hadoop-hdds-server-scm.classpath
-
-
- org.apache.hadoop
- hadoop-hdds-tools
- ${hdds.version}
- classpath
- cp
- hadoop-hdds-tools.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-s3gateway
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-s3gateway.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-csi
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-csi.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-ozone-manager
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-ozone-manager.classpath
-
-
-
- org.apache.hadoop
- hadoop-ozone-tools
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-tools.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-filesystem
- ${ozone.version}
- classpath
- cp
-
hadoop-ozone-filesystem.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-common
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-common.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-datanode
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-datanode.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-upgrade
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-upgrade.classpath
-
-
- org.apache.hadoop
- hadoop-ozone-insight
- ${ozone.version}
- classpath
- cp
- hadoop-ozone-insight.classpath
-
-
+ *.classpath
Review comment:
Fix me if I'm wrong, but `ozone classpath` now lists several modules
that were previously omitted.
Previously:
```
$ ozone classpath
Usage: ozone classpath
Where the artifact name is one of:
hadoop-hdds-server-scm
hadoop-hdds-tools
hadoop-ozone-common
hadoop-ozone-csi
hadoop-ozone-datanode
hadoop-ozone-filesystem
hadoop-ozone-insight
hadoop-ozone-ozone-manager
hadoop-ozone-recon
hadoop-ozone-s3gateway
hadoop-ozone-tools
hadoop-ozone-upgrade
```
With patch:
```
$ ozone classpath
Usage: ozone classpath
Where the artifact name is one of:
hadoop-hdds-client
hadoop-hdds-common
hadoop-hdds-config
hadoop-hdds-container-service
hadoop-hdds-docs
hadoop-hdds-hadoop-dependency-client
hadoop-hdds-hadoop-dependency-server
hadoop-hdds-interface-admin
hadoop-hdds-interface-client
hadoop-hdds-interface-server
hadoop-hdds-server-framework
hadoop-hdds-server-scm
hadoop-hdds-tools
hadoop-ozone-client
hadoop-ozone-common
hadoop-ozone-csi
hadoop-ozone-datanode
hadoop-ozone-filesystem-common
hadoop-ozone-filesystem
hadoop-ozone-insight
hadoop-ozone-interface-client
hadoop-ozone-ozone-manager
hadoop-ozone-recon
hadoop-ozone-reconcodegen
hadoop-ozone-s3gateway
hadoop-ozone-tools
hadoop-ozone-upgrade
```
Invoking `ozone classpath` for the extra modules results in an error (which
is the same on `master`):
```
$ ozone classpath hadoop-hdds-config
Error: Could not find or load main class org.apache.hadoop.util.Classpath
Caused by:
[GitHub] [hadoop-ozone] elek commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.
elek commented on pull request #1110: URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-651683723 > So, you mean, we should keep s3g's behavior consistent with aws s3? Yes, I think we should follow the behavior of AWS. There are so many s3 tools: doing exactly the same will cause less problem. But I agree that NPE should be replaced with more meaningful errors. @maobaolong Do we have the same error with this patch as AWS? (@bharatviswa504 thanks the screenshot. I didn't know that postman can generate the aws signaure header. Very useful information, started to use postman again...) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2949) mkdir : store directory entries in a separate table
[ https://issues.apache.org/jira/browse/HDDS-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2949: - Labels: backward-incompatible pull-request-available (was: backward-incompatible) > mkdir : store directory entries in a separate table > --- > > Key: HDDS-2949 > URL: https://issues.apache.org/jira/browse/HDDS-2949 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Supratim Deka >Assignee: Rakesh Radhakrishnan >Priority: Major > Labels: backward-incompatible, pull-request-available > > As of HDDS-2940, all the directories from the path prefix get created as > entries in the key table. as per the namespace proposal attached to > HDDS-2939, directory entries need to be stored in a separate "directory" > table. Files will continue to be stored in the key table, which can be > thought of as the "file" table. > The advantage of a separate directory table is to make directory lookup more > efficient - the entire table would fit into memory for a typical file based > dataset. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #912: WIP Patch - HDDS-2949: store dir/key entries in separate tables - first patch onl…
elek commented on pull request #912: URL: https://github.com/apache/hadoop-ozone/pull/912#issuecomment-651679104 /pending WIP This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on a change in pull request #1107: HDDS-3807. Propagate raft log disks info to SCM from datanode.
elek commented on a change in pull request #1107:
URL: https://github.com/apache/hadoop-ozone/pull/1107#discussion_r447545366
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerSpi.java
##
@@ -83,4 +85,12 @@ default void removeGroup(HddsProtos.PipelineID pipelineId)
* @return list of report for each pipeline.
*/
List getPipelineReport();
+
+ /**
+ * Get storage report for the XceiverServer instance.
+ * @return list of report for each storage location.
+ */
+ default List getStorageReport() throws
IOException {
Review comment:
It seems to be a good time the remove XceiverServerSpi interface.
Originally we had two separated implementation to connect to the datanode.
Today we have only one. One interface is used between the client and the
datanode, and the other one between datanode and ratis (datanode). As this
example shows, the two interface shouldn't be the same.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1107: HDDS-3807. Propagate raft log disks info to SCM from datanode.
elek commented on pull request #1107: URL: https://github.com/apache/hadoop-ozone/pull/1107#issuecomment-651678810 Checkstyle violation seems to be reported. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1053: HDDS-3737. Avoid serialization between UUID and String
runzhiwang commented on a change in pull request #1053:
URL: https://github.com/apache/hadoop-ozone/pull/1053#discussion_r447544624
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java
##
@@ -301,14 +306,21 @@ public static Pipeline
getFromProtobuf(HddsProtos.Pipeline pipeline)
for (DatanodeDetailsProto member : pipeline.getMembersList()) {
nodes.add(DatanodeDetails.getFromProtoBuf(member));
}
+UUID leaderId = null;
+if (pipeline.hasLeaderID() &&
+StringUtils.isNotEmpty(pipeline.getLeaderID())) {
+ leaderId = UUID.fromString(pipeline.getLeaderID());
Review comment:
@xiaoyuyao Sorry for missing the comment. @elek Thanks for reminding.
> The improvement will be limited as fromProtoBuf to String will prefer
process UUID before ID128
Do you mean we should prefer ID128 before process UUID.fromString ? If so, I
think you are totally right, I will update it. Besides I will also remove
UUID.fromString in the future.
> And to protoBuf will need to add both String version of UUID and the
long/long version of uuid 128
This is for compatibility, in the future, I will remove String version of
UUID.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1053: HDDS-3737. Avoid serialization between UUID and String
runzhiwang commented on a change in pull request #1053:
URL: https://github.com/apache/hadoop-ozone/pull/1053#discussion_r447544624
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java
##
@@ -301,14 +306,21 @@ public static Pipeline
getFromProtobuf(HddsProtos.Pipeline pipeline)
for (DatanodeDetailsProto member : pipeline.getMembersList()) {
nodes.add(DatanodeDetails.getFromProtoBuf(member));
}
+UUID leaderId = null;
+if (pipeline.hasLeaderID() &&
+StringUtils.isNotEmpty(pipeline.getLeaderID())) {
+ leaderId = UUID.fromString(pipeline.getLeaderID());
Review comment:
@xiaoyuyao Sorry for missing the comment. @elek Thanks for reminding.
> And to protoBuf will need to add both String version of UUID and the
long/long version of uuid 128
This is for compatibility, in the future, I will remove String version of
UUID.
> The improvement will be limited as fromProtoBuf to String will prefer
process UUID before ID128
Do you mean we should prefer ID128 before process UUID.fromString ? If so, I
think you are totally right, I will update it. Besides I will also remove
UUID.fromString in the future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1053: HDDS-3737. Avoid serialization between UUID and String
runzhiwang commented on a change in pull request #1053:
URL: https://github.com/apache/hadoop-ozone/pull/1053#discussion_r447544624
##
File path:
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java
##
@@ -301,14 +306,21 @@ public static Pipeline
getFromProtobuf(HddsProtos.Pipeline pipeline)
for (DatanodeDetailsProto member : pipeline.getMembersList()) {
nodes.add(DatanodeDetails.getFromProtoBuf(member));
}
+UUID leaderId = null;
+if (pipeline.hasLeaderID() &&
+StringUtils.isNotEmpty(pipeline.getLeaderID())) {
+ leaderId = UUID.fromString(pipeline.getLeaderID());
Review comment:
@xiaoyuyao Sorry for missing the comment. @elek Thanks for reminding.
> And to protoBuf will need to add both String version of UUID and the
long/long version of uuid 128
This is for compatibility, in the future, I will remove String version of
UUID.
> The improvement will be limited as fromProtoBuf to String will prefer
process UUID before ID128
Do you mean we should prefer ID128 before process UUID.fromString ? If so, I
think you are totally right, I will update it. Besides I will also remove
UUID.fromString in the future.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3770) Improve getPipelines performance
[ https://issues.apache.org/jira/browse/HDDS-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3770. --- Fix Version/s: 0.6.0 Resolution: Fixed > Improve getPipelines performance > > > Key: HDDS-3770 > URL: https://issues.apache.org/jira/browse/HDDS-3770 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: runzhiwang >Assignee: runzhiwang >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Attachments: screenshot-1.png > > > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1066: HDDS-3770. Improve getPipelines performance
elek commented on pull request #1066: URL: https://github.com/apache/hadoop-ozone/pull/1066#issuecomment-651674170 Merging this as it has green build + comments from @xiaoyuyao addressed and @dineshchitlangia also approved it. (thanks the reviews). Thanks the contribution @runzhiwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1066: HDDS-3770. Improve getPipelines performance
elek merged pull request #1066: URL: https://github.com/apache/hadoop-ozone/pull/1066 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1041: HDDS-3725. Ozone sh volume client support quota option
elek commented on pull request #1041: URL: https://github.com/apache/hadoop-ozone/pull/1041#issuecomment-651672572 Adding a few paragraphs to the jira (or to the PR) would help to understand the goal. It seems to introduce new type of quota (count based instead of byte based) which is not clear from the description This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1041: HDDS-3725. Ozone sh volume client support quota option
elek commented on pull request #1041: URL: https://github.com/apache/hadoop-ozone/pull/1041#issuecomment-651672160 /pending comments from @timmylicheng are not addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3699) Change write chunk failure logging to ERROR level in BlockOutputStream
[ https://issues.apache.org/jira/browse/HDDS-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3699. --- Fix Version/s: 0.6.0 Resolution: Fixed > Change write chunk failure logging to ERROR level in BlockOutputStream > -- > > Key: HDDS-3699 > URL: https://issues.apache.org/jira/browse/HDDS-3699 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Shashikant Banerjee >Assignee: Sadanand Shenoy >Priority: Major > Labels: Performance, pull-request-available > Fix For: 0.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1006: HDDS-3699. Change write chunk failure logging level to ERROR in BlockOutputStream.
elek commented on pull request #1006: URL: https://github.com/apache/hadoop-ozone/pull/1006#issuecomment-651669592 Ups, we got green build, but we forgot to merge it. Sorry for that. Doing it now. Thanks for the contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek merged pull request #1006: HDDS-3699. Change write chunk failure logging level to ERROR in BlockOutputStream.
elek merged pull request #1006: URL: https://github.com/apache/hadoop-ozone/pull/1006 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1053: HDDS-3737. Avoid serialization between UUID and String
elek commented on pull request #1053: URL: https://github.com/apache/hadoop-ozone/pull/1053#issuecomment-651668950 /pending @runzhiwang can you please add answer to the comment? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1033: HDDS-3667. If we gracefully stop datanode it would be better to notify scm and r…
elek commented on pull request #1033: URL: https://github.com/apache/hadoop-ozone/pull/1033#issuecomment-651667340 /pending See questions from @maobaolong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1028: HDDS-3735. Improve SCM performance with 3.7% by remove unnecessary lock and unlock
elek commented on pull request #1028: URL: https://github.com/apache/hadoop-ozone/pull/1028#issuecomment-651659061 It's not just about the `updateState` method, but (for example) the `removeContainer` method. Without write lock the container might be removed from `ownerMap` and not from the `typeMap`, which can cause inconsistency. (I think, but fix me if I am wrong) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
