[jira] [Commented] (HDDS-2708) Translate docs to Chinese
[ https://issues.apache.org/jira/browse/HDDS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054312#comment-17054312 ] YiSheng Lien commented on HDDS-2708: [~iamabug] I mean the English doc. > Translate docs to Chinese > - > > Key: HDDS-2708 > URL: https://issues.apache.org/jira/browse/HDDS-2708 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: upgrade >Reporter: Xiang Zhang >Assignee: Xiang Zhang >Priority: Major > > According to > [https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors], > I understand that Chinese docs are needed. I am interested in this, could > somebody give me some advice to get started ? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #650: HDDS-3089. TestSCMNodeManager intermittent crash
bharatviswa504 merged pull request #650: HDDS-3089. TestSCMNodeManager intermittent crash URL: https://github.com/apache/hadoop-ozone/pull/650 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3089) TestSCMNodeManager intermittent crash
[
https://issues.apache.org/jira/browse/HDDS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-3089:
-
Fix Version/s: 0.6.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestSCMNodeManager intermittent crash
> -
>
> Key: HDDS-3089
> URL: https://issues.apache.org/jira/browse/HDDS-3089
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.0
>
> Attachments: hs_err_pid9082.log,
> org.apache.hadoop.hdds.scm.node.TestSCMNodeManager-output.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> TestSCMNodeManager crashed in one of the runs, although it passes usually:
> {code:title=https://github.com/apache/hadoop-ozone/pull/601/checks?check_run_id=471611827}
> [ERROR] Crashed tests:
> [ERROR] org.apache.hadoop.hdds.scm.node.TestSCMNodeManager
> {code}
> {code:title=hs_err_pid9082.log}
> siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr:
> 0x7f378cf6f340
> Stack: [0x7f37626fb000,0x7f37627fc000], sp=0x7f37627f9e48, free
> space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C 0x7f378cf6f340
> C [librocksdbjni3775377216204452319.so+0x2a05dd]
> rocksdb::DB::Delete(rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&)+0x4d
> C [librocksdbjni3775377216204452319.so+0x2a0641]
> rocksdb::DBImpl::Delete(rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&)+0x11
> C [librocksdbjni3775377216204452319.so+0x1a931a]
> rocksdb::DB::Delete(rocksdb::WriteOptions const&, rocksdb::Slice const&)+0xba
> C [librocksdbjni3775377216204452319.so+0x19f3e0]
> rocksdb_delete_helper(JNIEnv_*, rocksdb::DB*, rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int)+0x130
> C [librocksdbjni3775377216204452319.so+0x19f4a1]
> Java_org_rocksdb_RocksDB_delete__J_3BII+0x41
> j org.rocksdb.RocksDB.delete(J[BII)V+0
> j org.rocksdb.RocksDB.delete([B)V+13
> j org.apache.hadoop.hdds.utils.RocksDBStore.delete([B)V+9
> j
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(Lorg/apache/hadoop/hdds/scm/pipeline/PipelineID;)V+35
> j
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(Lorg/apache/hadoop/hdds/scm/pipeline/Pipeline;)V+27
> ...
> j
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler.destroyPipelines(Lorg/apache/hadoop/hdds/protocol/DatanodeDetails;)V+28
> j
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler.onMessage(Lorg/apache/hadoop/hdds/protocol/DatanodeDetails;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V+6
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create :
create key table entries for intermediate directories in the path
URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389259516
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -105,24 +120,37 @@ public static OMPathInfo verifyFilesInPath(
LOG.trace("verifyFiles in Path : " + "/" + volumeName
+ "/" + bucketName + "/" + keyName + ":" + result);
-return new OMPathInfo(missing, result, inheritAcls);
+// if file exists, so does the parent. Assert.
+Preconditions.checkState((result != OMDirectoryResult.FILE_EXISTS)
+|| (parentExists));
+
+return new OMPathInfo(missing, result, inheritAcls, parentExists);
}
keyPath = keyPath.getParent();
}
if (inheritAcls.isEmpty()) {
- String bucketKey = omMetadataManager.getBucketKey(volumeName,
bucketName);
- inheritAcls =
omMetadataManager.getBucketTable().get(bucketKey).getAcls();
+ String bucketKey = omMetadataManager.getBucketKey(volumeName,
+ bucketName);
+ inheritAcls = omMetadataManager.getBucketTable().get(bucketKey)
+ .getAcls();
LOG.trace("Acls inherited from bucket " + bucketName + " are : "
+ inheritAcls);
}
LOG.trace("verifyFiles in Path : " + volumeName + "/" + bucketName + "/"
+ keyName + ":" + result);
+// parentExists always true if we are creating inside root dir
+Preconditions.checkState(!pathIsRootDir(immediateParentPath)
+|| parentExists);
// Found no files/ directories in the given path.
-return new OMPathInfo(missing, OMDirectoryResult.NONE, inheritAcls);
+return new OMPathInfo(missing, OMDirectoryResult.NONE, inheritAcls,
+parentExists);
Review comment:
this is false always.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389257388 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java ## @@ -325,7 +304,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, .setOpenVersion(openVersion).build()) .setCmdType(Type.CreateFile); omClientResponse = new OMFileCreateResponse(omResponse.build(), - omKeyInfo, clientID); + omKeyInfo, missingParentInfos, clientID); Review comment: lets also add these to the cache as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create :
create key table entries for intermediate directories in the path
URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389259006
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileRequest.java
##
@@ -80,6 +91,7 @@ public static OMPathInfo verifyFilesInPath(
// Check if this is actual file or a file in the given path
if (dbKeyName.equals(fileNameFromDetails)) {
result = OMDirectoryResult.FILE_EXISTS;
+ parentExists = true;
Review comment:
This should be false as this is a file and not a directory
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create :
create key table entries for intermediate directories in the path
URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389253641
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -379,6 +358,65 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
return omClientResponse;
}
+ private void checkAllParentsExist(OzoneManager ozoneManager,
+ KeyArgs keyArgs,
+ OMFileRequest.OMPathInfo pathInfo) throws IOException {
+Boolean canBeCreated = false;
+String keyName = keyArgs.getKeyName();
+OMFileRequest.OMDirectoryResult omDirectoryResult =
+pathInfo.getDirectoryResult();
+
+if (ozoneManager.createPrefixEntries()) {
+ // if immediate parent exists, assume higher level directories exist.
+ if (pathInfo.getDirectParentExists()) {
+canBeCreated = true;
+ }
Review comment:
we can throw an exception here, as parent does not exsit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389257243 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java ## @@ -60,6 +68,22 @@ protected void addToDBBatch(OMMetadataManager omMetadataManager, omKeyInfo.getBucketName(), omKeyInfo.getKeyName(), openKeySessionID); omMetadataManager.getOpenKeyTable().putWithBatch(batchOperation, openKey, omKeyInfo); +/** + * Create parent directory entries during Key Create - do not wait Review comment: let's also follow the same order in DirectoryCreateResponse. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389256924 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java ## @@ -60,6 +68,22 @@ protected void addToDBBatch(OMMetadataManager omMetadataManager, omKeyInfo.getBucketName(), omKeyInfo.getKeyName(), openKeySessionID); omMetadataManager.getOpenKeyTable().putWithBatch(batchOperation, openKey, omKeyInfo); +/** + * Create parent directory entries during Key Create - do not wait Review comment: Ideally, I feel we should create a directory prefix before the key/file prefix. this improves the readability. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create :
create key table entries for intermediate directories in the path
URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389259742
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -379,6 +358,65 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
return omClientResponse;
}
+ private void checkAllParentsExist(OzoneManager ozoneManager,
+ KeyArgs keyArgs,
+ OMFileRequest.OMPathInfo pathInfo) throws IOException {
+Boolean canBeCreated = false;
+String keyName = keyArgs.getKeyName();
+OMFileRequest.OMDirectoryResult omDirectoryResult =
+pathInfo.getDirectoryResult();
+
+if (ozoneManager.createPrefixEntries()) {
+ // if immediate parent exists, assume higher level directories exist.
+ if (pathInfo.getDirectParentExists()) {
Review comment:
The check for this is same as missingparents as null ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create :
create key table entries for intermediate directories in the path
URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389253686
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -379,6 +358,65 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
return omClientResponse;
}
+ private void checkAllParentsExist(OzoneManager ozoneManager,
+ KeyArgs keyArgs,
+ OMFileRequest.OMPathInfo pathInfo) throws IOException {
+Boolean canBeCreated = false;
+String keyName = keyArgs.getKeyName();
+OMFileRequest.OMDirectoryResult omDirectoryResult =
+pathInfo.getDirectoryResult();
+
+if (ozoneManager.createPrefixEntries()) {
+ // if immediate parent exists, assume higher level directories exist.
+ if (pathInfo.getDirectParentExists()) {
+canBeCreated = true;
+ }
+} else {
+ String volumeName = keyArgs.getVolumeName();
+ String bucketName = keyArgs.getBucketName();
+
+ // We cannot create a file if complete parent directories don't exist.
+
+ // verifyFilesInPath, checks only the path and its parent directories.
+ // But there may be some keys below the given path. So this method
+ // checks them.
+
+ // Example:
+ // Existing keys in table
+ // a/b/c/d/e
+ // a/b/c/d/f
+ // a/b
+
+ // Take an example if given key to be created with isRecursive set
+ // to false is "a/b/c/e".
+ // There is no key in keyTable with the provided path.
+ // Check in case if there are keys exist in given path. (This can
+ // happen if keys are directly created using key requests.)
+
+ // We need to do this check only in the case of non-recursive, so
+ // not included the checks done in checkKeysUnderPath in
+ // verifyFilesInPath method, as that method is common method for
+ // directory and file create request. This also avoid's this
+ // unnecessary check which is not required for those cases.
+ if (omDirectoryResult == NONE ||
+ omDirectoryResult == DIRECTORY_EXISTS_IN_GIVENPATH) {
+canBeCreated =
+checkKeysUnderPath(ozoneManager.getMetadataManager(), volumeName,
+bucketName, keyName);
+ } else if (omDirectoryResult == FILE_EXISTS_IN_GIVENPATH) {
+canBeCreated = false;
Review comment:
The exception here should be something like file exists in the actual path.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create :
create key table entries for intermediate directories in the path
URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389253436
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java
##
@@ -379,6 +358,65 @@ public OMClientResponse
validateAndUpdateCache(OzoneManager ozoneManager,
return omClientResponse;
}
+ private void checkAllParentsExist(OzoneManager ozoneManager,
+ KeyArgs keyArgs,
+ OMFileRequest.OMPathInfo pathInfo) throws IOException {
+Boolean canBeCreated = false;
+String keyName = keyArgs.getKeyName();
+OMFileRequest.OMDirectoryResult omDirectoryResult =
+pathInfo.getDirectoryResult();
+
+if (ozoneManager.createPrefixEntries()) {
Review comment:
createPrefixEntries can we rename this as prefixEntriesEnabled?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389260017 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMDirectoryCreateRequest.java ## @@ -347,7 +371,7 @@ private OmKeyInfo createDirectoryKeyInfoWithACL(OzoneManager ozoneManager, .build(); } - private OmKeyInfo createDirectoryKeyInfo(OzoneManager ozoneManager, + private static OmKeyInfo createDirectoryKeyInfo(OzoneManager ozoneManager, Review comment: both createDirectoryKeyInfoWithACL & createDirectoryKeyInfo can be collapsed into one function. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3089) TestSCMNodeManager intermittent crash
[
https://issues.apache.org/jira/browse/HDDS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-3089:
---
Status: Patch Available (was: In Progress)
> TestSCMNodeManager intermittent crash
> -
>
> Key: HDDS-3089
> URL: https://issues.apache.org/jira/browse/HDDS-3089
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Attachments: hs_err_pid9082.log,
> org.apache.hadoop.hdds.scm.node.TestSCMNodeManager-output.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> TestSCMNodeManager crashed in one of the runs, although it passes usually:
> {code:title=https://github.com/apache/hadoop-ozone/pull/601/checks?check_run_id=471611827}
> [ERROR] Crashed tests:
> [ERROR] org.apache.hadoop.hdds.scm.node.TestSCMNodeManager
> {code}
> {code:title=hs_err_pid9082.log}
> siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr:
> 0x7f378cf6f340
> Stack: [0x7f37626fb000,0x7f37627fc000], sp=0x7f37627f9e48, free
> space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C 0x7f378cf6f340
> C [librocksdbjni3775377216204452319.so+0x2a05dd]
> rocksdb::DB::Delete(rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&)+0x4d
> C [librocksdbjni3775377216204452319.so+0x2a0641]
> rocksdb::DBImpl::Delete(rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&)+0x11
> C [librocksdbjni3775377216204452319.so+0x1a931a]
> rocksdb::DB::Delete(rocksdb::WriteOptions const&, rocksdb::Slice const&)+0xba
> C [librocksdbjni3775377216204452319.so+0x19f3e0]
> rocksdb_delete_helper(JNIEnv_*, rocksdb::DB*, rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int)+0x130
> C [librocksdbjni3775377216204452319.so+0x19f4a1]
> Java_org_rocksdb_RocksDB_delete__J_3BII+0x41
> j org.rocksdb.RocksDB.delete(J[BII)V+0
> j org.rocksdb.RocksDB.delete([B)V+13
> j org.apache.hadoop.hdds.utils.RocksDBStore.delete([B)V+9
> j
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(Lorg/apache/hadoop/hdds/scm/pipeline/PipelineID;)V+35
> j
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(Lorg/apache/hadoop/hdds/scm/pipeline/Pipeline;)V+27
> ...
> j
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler.destroyPipelines(Lorg/apache/hadoop/hdds/protocol/DatanodeDetails;)V+28
> j
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler.onMessage(Lorg/apache/hadoop/hdds/protocol/DatanodeDetails;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V+6
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request #623: HDDS-2941. file create : create key table entries for intermediate directories in the path
mukul1987 commented on a change in pull request #623: HDDS-2941. file create :
create key table entries for intermediate directories in the path
URL: https://github.com/apache/hadoop-ozone/pull/623#discussion_r389251781
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java
##
@@ -60,6 +68,22 @@ protected void addToDBBatch(OMMetadataManager
omMetadataManager,
omKeyInfo.getBucketName(), omKeyInfo.getKeyName(), openKeySessionID);
omMetadataManager.getOpenKeyTable().putWithBatch(batchOperation,
openKey, omKeyInfo);
+/**
+ * Create parent directory entries during Key Create - do not wait
+ * for Key Commit request.
+ * XXX handle stale directory entries.
+ */
+if (parentKeyInfos != null) {
+ for (OmKeyInfo parentKeyInfo : parentKeyInfos) {
+String parentKey = omMetadataManager
+.getOzoneDirKey(parentKeyInfo.getVolumeName(),
+parentKeyInfo.getBucketName(), parentKeyInfo.getKeyName());
+LOG.debug("putWithBatch adding parent : key {} info : {}", parentKey,
Review comment:
lets wrap around LOG.isDebugEnabled()
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #650: HDDS-3089. TestSCMNodeManager intermittent crash
adoroszlai opened a new pull request #650: HDDS-3089. TestSCMNodeManager intermittent crash URL: https://github.com/apache/hadoop-ozone/pull/650 ## What changes were proposed in this pull request? Based on the logs the test crashes during teardown when trying to delete pipeline in `EventQueue-DeadNodeForDeadNodeHandler` thread, as the RocksDB for pipelines was already closed during SCM stop. 1. Clear `db` reference in `RocksDBStore` after closing it, to avoid calls on closed RocksDB. This might result in NPE, which would be handled in Ozone normally, while call on closed DB might result in seg.fault in native code. 2. Protect `pipelineStore.close()` call with write lock in `SCMPipelineManager` to avoid concurrent `pipelineStore.delete()` (or other similar operations). https://issues.apache.org/jira/browse/HDDS-3089 ## How was this patch tested? Executed TestSCMNodeManager 2x20 times successfully: https://github.com/adoroszlai/hadoop-ozone/runs/491868681 https://github.com/adoroszlai/hadoop-ozone/runs/491888786 Regular CI with all checks (only it-freon failed): https://github.com/adoroszlai/hadoop-ozone/runs/491950482 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3089) TestSCMNodeManager intermittent crash
[
https://issues.apache.org/jira/browse/HDDS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-3089:
-
Labels: pull-request-available (was: )
> TestSCMNodeManager intermittent crash
> -
>
> Key: HDDS-3089
> URL: https://issues.apache.org/jira/browse/HDDS-3089
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
> Attachments: hs_err_pid9082.log,
> org.apache.hadoop.hdds.scm.node.TestSCMNodeManager-output.txt
>
>
> TestSCMNodeManager crashed in one of the runs, although it passes usually:
> {code:title=https://github.com/apache/hadoop-ozone/pull/601/checks?check_run_id=471611827}
> [ERROR] Crashed tests:
> [ERROR] org.apache.hadoop.hdds.scm.node.TestSCMNodeManager
> {code}
> {code:title=hs_err_pid9082.log}
> siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr:
> 0x7f378cf6f340
> Stack: [0x7f37626fb000,0x7f37627fc000], sp=0x7f37627f9e48, free
> space=1019k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> C 0x7f378cf6f340
> C [librocksdbjni3775377216204452319.so+0x2a05dd]
> rocksdb::DB::Delete(rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&)+0x4d
> C [librocksdbjni3775377216204452319.so+0x2a0641]
> rocksdb::DBImpl::Delete(rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&)+0x11
> C [librocksdbjni3775377216204452319.so+0x1a931a]
> rocksdb::DB::Delete(rocksdb::WriteOptions const&, rocksdb::Slice const&)+0xba
> C [librocksdbjni3775377216204452319.so+0x19f3e0]
> rocksdb_delete_helper(JNIEnv_*, rocksdb::DB*, rocksdb::WriteOptions const&,
> rocksdb::ColumnFamilyHandle*, _jbyteArray*, int, int)+0x130
> C [librocksdbjni3775377216204452319.so+0x19f4a1]
> Java_org_rocksdb_RocksDB_delete__J_3BII+0x41
> j org.rocksdb.RocksDB.delete(J[BII)V+0
> j org.rocksdb.RocksDB.delete([B)V+13
> j org.apache.hadoop.hdds.utils.RocksDBStore.delete([B)V+9
> j
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.removePipeline(Lorg/apache/hadoop/hdds/scm/pipeline/PipelineID;)V+35
> j
> org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.destroyPipeline(Lorg/apache/hadoop/hdds/scm/pipeline/Pipeline;)V+27
> ...
> j
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler.destroyPipelines(Lorg/apache/hadoop/hdds/protocol/DatanodeDetails;)V+28
> j
> org.apache.hadoop.hdds.scm.node.DeadNodeHandler.onMessage(Lorg/apache/hadoop/hdds/protocol/DatanodeDetails;Lorg/apache/hadoop/hdds/server/events/EventPublisher;)V+6
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
