[jira] [Assigned] (HDDS-363) Faster datanode registration during the first startup
[ https://issues.apache.org/jira/browse/HDDS-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-363: - Assignee: Elek, Marton > Faster datanode registration during the first startup > - > > Key: HDDS-363 > URL: https://issues.apache.org/jira/browse/HDDS-363 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > > During the first startup usually we need to wait about 30 s to find the scm > usable. The datanode registration is a multiple step process > (request/response + request/response) and we need to wait the next HB to > finish the registration. > I propose to use a more higher HB frequency at startup (let's say 2 seconds) > and set the configured HB only at the end of the registration. > It also helps for the first users as it could be less confusing (the datanode > can be seen almost immediately on the UI) > Also it would help a lot for me during the testing (yes, I can decrease the > HB frequency but in that case it's harder the follow the later HBs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-222) Remove hdfs command line from ozone distrubution.
[
https://issues.apache.org/jira/browse/HDDS-222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588500#comment-16588500
]
Elek, Marton commented on HDDS-222:
---
I prefer to change all the dependencies to compile level unless we have a
strong reason to keep it on provided. According to the maven nomenclature they
should be compile scope anyway. Provided was used just as a workaround (IMHO).
According to this page:
https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
{quote}provided: This is much like compile, but indicates you expect the JDK
or a container to provide the dependency at runtime. For example, when building
a web application for the Java Enterprise Edition, you would set the dependency
on the Servlet API and related Java EE APIs to scope provided because the web
container provides those classes{quote}
I think it's not our use case. We need the dependencies and the jars should be
there.
Compile time dependency also helps a lot to start the service components from
IDE.
> Remove hdfs command line from ozone distrubution.
> -
>
> Key: HDDS-222
> URL: https://issues.apache.org/jira/browse/HDDS-222
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: newbie
> Fix For: 0.2.1
>
> Attachments: HDDS-222.001.patch, HDDS-222.002.patch
>
>
> As the ozone release artifact doesn't contain a stable namenode/datanode code
> the hdfs command should be removed from the ozone artifact.
> ozone-dist-layout-stitching also could be simplified to copy only the
> required jar files (we don't need to copy the namenode/datanode server side
> jars, just the common artifacts
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-328) Support export and import of the KeyValueContainer
[
https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588586#comment-16588586
]
Elek, Marton commented on HDDS-328:
---
This is a false positive warning and I didn't find any workaround.
Locally I got the following error:
{code}
[INFO] --- findbugs-maven-plugin:3.0.0:check (default-cli) @
hadoop-hdds-container-service ---
[INFO] BugInstance size is 1
[INFO] Error size is 0
[INFO] Total bugs: 1
[INFO] Possible null pointer dereference in
org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.extractEntry(TarArchiveInputStream,
long, Path) due to return value of called method
["org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker"] At
TarContainerPacker.java:[lines 52-247]
{code}
It's hard to say what's the problem as lines 52-247 is a huge range.
After a while I found that the problem is with the Files.createDirectories.
It's passed with commenting out the Files.createDirectories line. But path and
path.getParent is checked before with preconditions check.
{code}
private void extractEntry(TarArchiveInputStream tarInput, long size,
Path path) throws IOException {
Preconditions.checkNotNull(path, "Path element should not be null");
Preconditions.checkNotNull(path.getParent(), "Path element should "
+ "have a parent directory");
//Files.createDirectories(path.getParent()); // It's ok without this line
(wat?)
try (BufferedOutputStream bos = new BufferedOutputStream(
new FileOutputStream(path.toAbsolutePath().toString( {
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize + 1];
long remaining = size;
while (remaining > 0) {
int read =
tarInput.read(buffer, 0, (int) Math.min(remaining, bufferSize));
if (read >= 0) {
remaining -= read;
bos.write(buffer, 0, read);
} else {
remaining = 0;
}
}
}
}
{code}
Finally I added the issue to the findbugsExclude file:
{code}
diff --git a/hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
b/hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
index 3571a8929e3..861d28d6dfb 100644
--- a/hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
+++ b/hadoop-hdds/container-service/dev-support/findbugsExcludeFile.xml
@@ -18,4 +18,9 @@
+
+
+
+
+
{code}
> Support export and import of the KeyValueContainer
> --
>
> Key: HDDS-328
> URL: https://issues.apache.org/jira/browse/HDDS-328
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Critical
> Fix For: 0.2.1
>
> Attachments: HDDS-328-HDFS-10285.006.patch, HDDS-328.002.patch,
> HDDS-328.003.patch, HDDS-328.004.patch, HDDS-328.005.patch, HDDS-328.007.patch
>
>
> In HDDS-75 we pack the container data to an archive file, copy to other
> datanodes and create the container from the archive.
> As I wrote in the comment of HDDS-75 I propose to separate the patch to make
> it easier to review.
> In this patch we need to extend the existing Container interface with adding
> export/import methods to save the container data to one binary input/output
> stream.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-328) Support export and import of the KeyValueContainer
[ https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-328: -- Attachment: HDDS-328.007.patch > Support export and import of the KeyValueContainer > -- > > Key: HDDS-328 > URL: https://issues.apache.org/jira/browse/HDDS-328 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Critical > Fix For: 0.2.1 > > Attachments: HDDS-328-HDFS-10285.006.patch, HDDS-328.002.patch, > HDDS-328.003.patch, HDDS-328.004.patch, HDDS-328.005.patch, HDDS-328.007.patch > > > In HDDS-75 we pack the container data to an archive file, copy to other > datanodes and create the container from the archive. > As I wrote in the comment of HDDS-75 I propose to separate the patch to make > it easier to review. > In this patch we need to extend the existing Container interface with adding > export/import methods to save the container data to one binary input/output > stream. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-328) Support export and import of the KeyValueContainer
[ https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-328: -- Attachment: HDDS-328.008.patch > Support export and import of the KeyValueContainer > -- > > Key: HDDS-328 > URL: https://issues.apache.org/jira/browse/HDDS-328 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Critical > Fix For: 0.2.1 > > Attachments: HDDS-328-HDFS-10285.006.patch, HDDS-328.002.patch, > HDDS-328.003.patch, HDDS-328.004.patch, HDDS-328.005.patch, > HDDS-328.007.patch, HDDS-328.008.patch > > > In HDDS-75 we pack the container data to an archive file, copy to other > datanodes and create the container from the archive. > As I wrote in the comment of HDDS-75 I propose to separate the patch to make > it easier to review. > In this patch we need to extend the existing Container interface with adding > export/import methods to save the container data to one binary input/output > stream. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-325) Add event watcher for delete blocks command
[ https://issues.apache.org/jira/browse/HDDS-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590357#comment-16590357 ] Elek, Marton commented on HDDS-325: --- Thanks you very much [~ljain] the patch. It's much more simple and clean for me. * Over it looks good to me, but I am not very familiar with the deletion part. Additional comments would be good from somebody else. As far as I understood it's good. * Adding ContainerBlocksDeletionACKProto to the status field looks strange to me, but to be honest I have no better idea. I guess with proto 3 we will have better tools to use OOP polymorphism. * I am not sure but I think in RetriableDatanodeEventWatcher.onTimeout we need to send the message to SCMEvents.DATANODE_COMMAND and not SCMEvents.RETRIABLE_DATANODE_COMMAND (A unit test would help to decide this question...) * About EventWatcher. I don't understand exactly your example. I think one EventWatcher (or more preciously one RetriableDatanodeEventWatcher) would be enough. Let's say we have two kind of commands : new CommandForDatanode<>(dnId, new DeleteBlocksCommand) new CommandForDatanode<>(dnId, new EatBananaCommand) Both could be sent to the SCMEvents.RETRIABLE_DATANODE_COMMAND for RetriableDatanodeEventWatcher (and for the scmNodeManager) and they could handle both of them. With listening on the CMD_STATUS_REPORT event RetriableDatanodeEventWatcher could handle all of the retries in one place. More formerly I have two problems with the changes in EventWatcher: 1) we don't need it in this patch (the new EventWatcher.watchEvents method is unused). I would add it when it required. 2) I think the main problem is that the addHandler calls are part of the EventWatcher (this is my fault, but since the original creation all of the wiring logic moved to the StorageContainerManager). I think we can either remove the wiring logic from the EventHandler and move to the StorageContainerManager (where we can easily add EventWatcher as a listener of multiple events) or we can create a builder (EventHandler.watchEvents is almost like a builder). Or both! But I propose to do it in a separated jira (as it's not strictly required now as I understood, the watchEvents is still unused) One minor nit: I would follow the naming convention in SCMEvents: L92 (using underscores). > Add event watcher for delete blocks command > --- > > Key: HDDS-325 > URL: https://issues.apache.org/jira/browse/HDDS-325 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-325.001.patch, HDDS-325.002.patch, > HDDS-325.003.patch, HDDS-325.004.patch, HDDS-325.005.patch > > > This Jira aims to add watcher for deleteBlocks command. It removes the > current rpc call required for datanode to send the acknowledgement for > deleteBlocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-328) Support export and import of the KeyValueContainer
[ https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-328: -- Attachment: HDDS-328.010.patch > Support export and import of the KeyValueContainer > -- > > Key: HDDS-328 > URL: https://issues.apache.org/jira/browse/HDDS-328 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Critical > Fix For: 0.2.1 > > Attachments: HDDS-328-HDFS-10285.006.patch, HDDS-328.002.patch, > HDDS-328.003.patch, HDDS-328.004.patch, HDDS-328.005.patch, > HDDS-328.007.patch, HDDS-328.008.patch, HDDS-328.009.patch, HDDS-328.010.patch > > > In HDDS-75 we pack the container data to an archive file, copy to other > datanodes and create the container from the archive. > As I wrote in the comment of HDDS-75 I propose to separate the patch to make > it easier to review. > In this patch we need to extend the existing Container interface with adding > export/import methods to save the container data to one binary input/output > stream. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-364) Update open container replica information in SCM during DN register
[ https://issues.apache.org/jira/browse/HDDS-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590540#comment-16590540 ] Elek, Marton commented on HDDS-364: --- Hmm. I am not sure. Locally TestContainerMapping was passed for me without this patch but failed with the patch. Seems to be related. Could you please check if it's a real problem? > Update open container replica information in SCM during DN register > --- > > Key: HDDS-364 > URL: https://issues.apache.org/jira/browse/HDDS-364 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-364.00.patch, HDDS-364.01.patch > > > Update open container replica information in SCM during DN register. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-364) Update open container replica information in SCM during DN register
[ https://issues.apache.org/jira/browse/HDDS-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590514#comment-16590514 ] Elek, Marton commented on HDDS-364: --- Thanks the answers [~ajayydv]. I agree with you. It would be great to consolidate all the container report processing part to have just one method instead of the multiple smaller one. But it's definitely a bigger effort and requires a separated Jira and discussion. I understood that this is just enough small change to get the replication numbers for calculating the chill mode exit criteria. From this point of view it looks good to me. Will double check if the test failures are unrelated and commit it if no other objections by anybody else... > Update open container replica information in SCM during DN register > --- > > Key: HDDS-364 > URL: https://issues.apache.org/jira/browse/HDDS-364 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-364.00.patch, HDDS-364.01.patch > > > Update open container replica information in SCM during DN register. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-328) Support export and import of the KeyValueContainer
[ https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590378#comment-16590378 ] Elek, Marton commented on HDDS-328: --- WOW. Didn't know that the Preconditions checks have return values. Thank you to show to me. I reverted the findbugs/exclude change and did what you suggested. Will upload the patch in the next minutes after the final local test. > Support export and import of the KeyValueContainer > -- > > Key: HDDS-328 > URL: https://issues.apache.org/jira/browse/HDDS-328 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Critical > Fix For: 0.2.1 > > Attachments: HDDS-328-HDFS-10285.006.patch, HDDS-328.002.patch, > HDDS-328.003.patch, HDDS-328.004.patch, HDDS-328.005.patch, > HDDS-328.007.patch, HDDS-328.008.patch, HDDS-328.009.patch > > > In HDDS-75 we pack the container data to an archive file, copy to other > datanodes and create the container from the archive. > As I wrote in the comment of HDDS-75 I propose to separate the patch to make > it easier to review. > In this patch we need to extend the existing Container interface with adding > export/import methods to save the container data to one binary input/output > stream. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591312#comment-16591312 ] Elek, Marton commented on HDDS-359: --- [~anu] Yetus tries to minimalize the number of unit tests. It checks the changed maven projects and tries to calculate the required tests. If ozone-integration project is not changed, the tests won't be executed. To modify this behaviour we need to adjust the hadoop yetus presonality (which is stored in the yetus repository). > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch, > HDDS-359.003.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-373) Ozone genconf tool must generate ozone-site.xml with sample values instead of a template
[ https://issues.apache.org/jira/browse/HDDS-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591428#comment-16591428 ] Elek, Marton commented on HDDS-373: --- Thanks [~dineshchitlangia] the patch. This patch not only change the output of getconf utility but changes the default values. I was not part of the original discussion but it also means that the configuration values are not required any more. If that was the intention some additional checks (such as in OmUtils.java: L69) may be removed. I have no opinion about what is the best: provide a localhost default or require to set the configuration. [~anu] can comment it. Also /data/disk1/meta is not a reasonable default IMHO. /tmp/disk1/meta would be better. (Or even better generating a value based on java.io.tmpdir to make our osx/windows friends also happy) > Ozone genconf tool must generate ozone-site.xml with sample values instead of > a template > > > Key: HDDS-373 > URL: https://issues.apache.org/jira/browse/HDDS-373 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Tools >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Attachments: HDDS-373.001.patch > > > As discussed with [~anu], currently, the genconf tool generates a template > ozone-site.xml. This is not very useful for new users as they would have to > understand what values should be set for the minimal configuration properties. > This Jira proposes to modify the ozone-default.xml which is leveraged by > genconf tool to generate ozone-site.xml -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-334) Update GettingStarted page to mention details about Ozone GenConf tool
[
https://issues.apache.org/jira/browse/HDDS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591347#comment-16591347
]
Elek, Marton commented on HDDS-334:
---
Thanks [~dineshchitlangia] the change. It looks good to me.
Not a blocker, but I would consider to move this information earlier in the
document.
{code}
## Create ozone-site.xml
Ozone relies on its own configuration file called ozone-site.xml.
__TO HERE__
The following are the most important settings.
...
{code}
It may be more easy to find the tool before the "create configuration" step.
> Update GettingStarted page to mention details about Ozone GenConf tool
> --
>
> Key: HDDS-334
> URL: https://issues.apache.org/jira/browse/HDDS-334
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
>Reporter: Dinesh Chitlangia
>Assignee: Dinesh Chitlangia
>Priority: Major
> Labels: documentation
> Attachments: HDDS-334.001.patch
>
>
> Add description about Ozone GenConf tool in GettingStarted page
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-369) Remove the containers of a dead node from the container state map
[ https://issues.apache.org/jira/browse/HDDS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591655#comment-16591655 ] Elek, Marton commented on HDDS-369: --- Thanks [~hanishakoneru] to check it. The shiny new patch is just uploaded. Let's see what jenkins will say... > Remove the containers of a dead node from the container state map > - > > Key: HDDS-369 > URL: https://issues.apache.org/jira/browse/HDDS-369 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-369.001.patch, HDDS-369.002.patch > > > In case of a node is dead we need to update the container replicas > information of the containerStateMap for all the containers from that > specific node. > With removing the replica information we can detect the under replicated > state and start the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-369) Remove the containers of a dead node from the container state map
[ https://issues.apache.org/jira/browse/HDDS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-369: -- Attachment: HDDS-369.002.patch > Remove the containers of a dead node from the container state map > - > > Key: HDDS-369 > URL: https://issues.apache.org/jira/browse/HDDS-369 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-369.001.patch, HDDS-369.002.patch > > > In case of a node is dead we need to update the container replicas > information of the containerStateMap for all the containers from that > specific node. > With removing the replica information we can detect the under replicated > state and start the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591672#comment-16591672
]
Elek, Marton commented on HDDS-280:
---
Pseudo cluster files are fixed with HDDS-218, acceptance tests are fixed in
this patch + symbolic link creation is removed.
Thanks all the discussions, please review this approach.
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-280.001.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-280:
--
Status: Patch Available (was: Open)
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-280.001.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-363) Faster datanode registration during the first startup
[ https://issues.apache.org/jira/browse/HDDS-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-363: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Faster datanode registration during the first startup > - > > Key: HDDS-363 > URL: https://issues.apache.org/jira/browse/HDDS-363 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-363.001.patch, HDDS-363.002.patch > > > During the first startup usually we need to wait about 30 s to find the scm > usable. The datanode registration is a multiple step process > (request/response + request/response) and we need to wait the next HB to > finish the registration. > I propose to use a more higher HB frequency at startup (let's say 2 seconds) > and set the configured HB only at the end of the registration. > It also helps for the first users as it could be less confusing (the datanode > can be seen almost immediately on the UI) > Also it would help a lot for me during the testing (yes, I can decrease the > HB frequency but in that case it's harder the follow the later HBs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-363) Faster datanode registration during the first startup
[ https://issues.apache.org/jira/browse/HDDS-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591677#comment-16591677 ] Elek, Marton commented on HDDS-363: --- Thanks the review [~hanishakoneru]. Will commit it to the trunk soon > Faster datanode registration during the first startup > - > > Key: HDDS-363 > URL: https://issues.apache.org/jira/browse/HDDS-363 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-363.001.patch, HDDS-363.002.patch > > > During the first startup usually we need to wait about 30 s to find the scm > usable. The datanode registration is a multiple step process > (request/response + request/response) and we need to wait the next HB to > finish the registration. > I propose to use a more higher HB frequency at startup (let's say 2 seconds) > and set the configured HB only at the end of the registration. > It also helps for the first users as it could be less confusing (the datanode > can be seen almost immediately on the UI) > Also it would help a lot for me during the testing (yes, I can decrease the > HB frequency but in that case it's harder the follow the later HBs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-227) Use Grpc as the default transport protocol for Standalone pipeline
[ https://issues.apache.org/jira/browse/HDDS-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591603#comment-16591603 ] Elek, Marton commented on HDDS-227: --- Thanks the updated. Could you please check the failing test. I checked it locally and it should be related. Without the patch it's passing, but with the patch it's failing. > Use Grpc as the default transport protocol for Standalone pipeline > -- > > Key: HDDS-227 > URL: https://issues.apache.org/jira/browse/HDDS-227 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-227.001.patch, HDDS-227.002.patch, > HDDS-227.003.patch > > > Using a config, Standalone pipeline can currently choose between Grpc and > Netty based transport protocol, this jira proposes to use only grpc as the > transport protocol. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-374) Support to configure container size in units lesser than GB
[ https://issues.apache.org/jira/browse/HDDS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591741#comment-16591741 ] Elek, Marton commented on HDDS-374: --- +1. It would help a lot during the local testing. > Support to configure container size in units lesser than GB > --- > > Key: HDDS-374 > URL: https://issues.apache.org/jira/browse/HDDS-374 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Fix For: 0.2.1 > > > After HDDS-317 we can configure the container size with its unit (eg. 5gb, > 1000mb etc.). But we still require it to be in multiples of GB, the > configured value will be rounded off (floor) to the nearest GB value. It will > be helpful to have support for units lesser than GB, it will make our life > simpler while writing unit tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-276) Fix symbolic link creation during Ozone dist process
[ https://issues.apache.org/jira/browse/HDDS-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton resolved HDDS-276. --- Resolution: Won't Fix Will be fixed by HDDS-280 > Fix symbolic link creation during Ozone dist process > > > Key: HDDS-276 > URL: https://issues.apache.org/jira/browse/HDDS-276 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.2.1 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-276.001.patch > > > Ozone is creating a symlink during the dist process. > Using the "ozone" directory as a destination name all the docker-based > acceptance tests and docker-compose files are more simple as they don't need > to have the version information in the path. > But to keep the version specific folder name in the tar file we create a > symbolic link during the tar creation. With the symbolic link and the > '–dereference' tar argument we can create the tar file which includes a > versioned directory (ozone-0.2.1) but we can use the a dist directory without > the version in the name (hadoop-dist/target/ozone). > Currently this symlink creation has an issue. It couldn't be run twice. You > need to do a 'mvn clean' before you can create a new dist. > But fortunately this could be fixed easily by checking if the destination > symlink exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-280:
--
Attachment: HDDS-280.001.patch
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-280.001.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton reassigned HDDS-280:
-
Assignee: Elek, Marton
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-280.001.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-334) Update GettingStarted page to mention details about Ozone GenConf tool
[ https://issues.apache.org/jira/browse/HDDS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591681#comment-16591681 ] Elek, Marton commented on HDDS-334: --- Fine with me. (I would like to just give more light to the tool. Only a few person scroll until the (almost) end of the page...) > Update GettingStarted page to mention details about Ozone GenConf tool > -- > > Key: HDDS-334 > URL: https://issues.apache.org/jira/browse/HDDS-334 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: document >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: documentation > Attachments: HDDS-334.001.patch > > > Add description about Ozone GenConf tool in GettingStarted page -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-364) Update open container replica information in SCM during DN register
[ https://issues.apache.org/jira/browse/HDDS-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592136#comment-16592136 ] Elek, Marton commented on HDDS-364: --- +1. The last test failure is (AFAIK) not related. It is some Xceiver related timeout error (with some strange fork error). This part has not been changed in this patch. Will commit it to the trunk shortly. > Update open container replica information in SCM during DN register > --- > > Key: HDDS-364 > URL: https://issues.apache.org/jira/browse/HDDS-364 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-364.00.patch, HDDS-364.01.patch, HDDS-364.02.patch, > HDDS-364.03.patch > > > Update open container replica information in SCM during DN register. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-364) Update open container replica information in SCM during DN register
[ https://issues.apache.org/jira/browse/HDDS-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-364: -- Resolution: Fixed Status: Resolved (was: Patch Available) Just landed on trunk. Thank you [~ajayydv] the contribution. > Update open container replica information in SCM during DN register > --- > > Key: HDDS-364 > URL: https://issues.apache.org/jira/browse/HDDS-364 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-364.00.patch, HDDS-364.01.patch, HDDS-364.02.patch, > HDDS-364.03.patch > > > Update open container replica information in SCM during DN register. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-313) Add metrics to containerState Machine
[ https://issues.apache.org/jira/browse/HDDS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592122#comment-16592122 ] Elek, Marton commented on HDDS-313: --- More realistic results. Can [~candychencan] please check the unit test failure and the findbug error? > Add metrics to containerState Machine > - > > Key: HDDS-313 > URL: https://issues.apache.org/jira/browse/HDDS-313 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-313.001.patch, HDDS-313.002.patch, > HDDS-313.003.patch, HDDS-313.004.patch, HDDS-313.005.patch, HDDS-313.006.patch > > > metrics needs to be added to containerStateMachine to keep track of various > ratis ops like writeStateMachine/readStateMachine/applyTransactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-313) Add metrics to containerState Machine
[ https://issues.apache.org/jira/browse/HDDS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-313: -- Attachment: HDDS-313.006.patch > Add metrics to containerState Machine > - > > Key: HDDS-313 > URL: https://issues.apache.org/jira/browse/HDDS-313 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-313.001.patch, HDDS-313.002.patch, > HDDS-313.003.patch, HDDS-313.004.patch, HDDS-313.005.patch, HDDS-313.006.patch > > > metrics needs to be added to containerStateMachine to keep track of various > ratis ops like writeStateMachine/readStateMachine/applyTransactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-313) Add metrics to containerState Machine
[ https://issues.apache.org/jira/browse/HDDS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591554#comment-16591554 ] Elek, Marton commented on HDDS-313: --- Reuploaded the v5 patch with a new name to get latest jenkins results. Some unit tests are failed (including the new one which is introduced by this patch). But according to the logs it could be a jenkins failure. (connection timeouts...) > Add metrics to containerState Machine > - > > Key: HDDS-313 > URL: https://issues.apache.org/jira/browse/HDDS-313 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-313.001.patch, HDDS-313.002.patch, > HDDS-313.003.patch, HDDS-313.004.patch, HDDS-313.005.patch, HDDS-313.006.patch > > > metrics needs to be added to containerStateMachine to keep track of various > ratis ops like writeStateMachine/readStateMachine/applyTransactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-317) Use new StorageSize API for reading ozone.scm.container.size.gb
[
https://issues.apache.org/jira/browse/HDDS-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591564#comment-16591564
]
Elek, Marton commented on HDDS-317:
---
+1. Looks good to me.
I executed a full acceptance test and everything worked well.
Will commit it to the trunk shortly. (there were 3 remaining 'unused imports'
errors, I just removed them during the test).
> Use new StorageSize API for reading ozone.scm.container.size.gb
> ---
>
> Key: HDDS-317
> URL: https://issues.apache.org/jira/browse/HDDS-317
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Nanda kumar
>Assignee: Junjie Chen
>Priority: Major
> Labels: newbie
> Fix For: 0.2.1
>
> Attachments: HDDS-317.2.patch, HDDS-317.3.patch, HDDS-317.4.patch,
> HDDS-317.patch
>
>
> Container size is configured using property {{ozone.scm.container.size.gb}}.
> This can be renamed to {{ozone.scm.container.size}} and use new StorageSize
> API to read the value.
> The property is defined in
> 1. ozone-default.xml
> 2. ScmConfigKeys#OZONE_SCM_CONTAINER_SIZE_GB
> The default value is defined in
> 1. ozone-default.xml
> 2. {{ScmConfigKeys#OZONE_SCM_CONTAINER_SIZE_DEFAULT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-317) Use new StorageSize API for reading ozone.scm.container.size.gb
[
https://issues.apache.org/jira/browse/HDDS-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-317:
--
Attachment: HDDS-317.005.patch
> Use new StorageSize API for reading ozone.scm.container.size.gb
> ---
>
> Key: HDDS-317
> URL: https://issues.apache.org/jira/browse/HDDS-317
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Nanda kumar
>Assignee: Junjie Chen
>Priority: Major
> Labels: newbie
> Fix For: 0.2.1
>
> Attachments: HDDS-317.005.patch, HDDS-317.2.patch, HDDS-317.3.patch,
> HDDS-317.4.patch, HDDS-317.patch
>
>
> Container size is configured using property {{ozone.scm.container.size.gb}}.
> This can be renamed to {{ozone.scm.container.size}} and use new StorageSize
> API to read the value.
> The property is defined in
> 1. ozone-default.xml
> 2. ScmConfigKeys#OZONE_SCM_CONTAINER_SIZE_GB
> The default value is defined in
> 1. ozone-default.xml
> 2. {{ScmConfigKeys#OZONE_SCM_CONTAINER_SIZE_DEFAULT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-317) Use new StorageSize API for reading ozone.scm.container.size.gb
[
https://issues.apache.org/jira/browse/HDDS-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-317:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to the trunk. Thank you [~junjie] the contribution.
> Use new StorageSize API for reading ozone.scm.container.size.gb
> ---
>
> Key: HDDS-317
> URL: https://issues.apache.org/jira/browse/HDDS-317
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Nanda kumar
>Assignee: Junjie Chen
>Priority: Major
> Labels: newbie
> Fix For: 0.2.1
>
> Attachments: HDDS-317.005.patch, HDDS-317.2.patch, HDDS-317.3.patch,
> HDDS-317.4.patch, HDDS-317.patch
>
>
> Container size is configured using property {{ozone.scm.container.size.gb}}.
> This can be renamed to {{ozone.scm.container.size}} and use new StorageSize
> API to read the value.
> The property is defined in
> 1. ozone-default.xml
> 2. ScmConfigKeys#OZONE_SCM_CONTAINER_SIZE_GB
> The default value is defined in
> 1. ozone-default.xml
> 2. {{ScmConfigKeys#OZONE_SCM_CONTAINER_SIZE_DEFAULT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-222) Remove hdfs command line from ozone distrubution.
[ https://issues.apache.org/jira/browse/HDDS-222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-222: -- Attachment: HDDS-222.003.patch > Remove hdfs command line from ozone distrubution. > - > > Key: HDDS-222 > URL: https://issues.apache.org/jira/browse/HDDS-222 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie > Fix For: 0.2.1 > > Attachments: HDDS-222.001.patch, HDDS-222.002.patch, > HDDS-222.003.patch > > > As the ozone release artifact doesn't contain a stable namenode/datanode code > the hdfs command should be removed from the ozone artifact. > ozone-dist-layout-stitching also could be simplified to copy only the > required jar files (we don't need to copy the namenode/datanode server side > jars, just the common artifacts -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595094#comment-16595094
]
Elek, Marton commented on HDDS-343:
---
Sure, I just uploaded the rebased version.
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch, HDDS-343.002.patch,
> HDDS-343.003.patch, HDDS-343.004.patch, HDDS-343.005.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be closed.
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 2
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 2 is in CLOSING state and need not be closed.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-343:
--
Attachment: HDDS-343.005.patch
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch, HDDS-343.002.patch,
> HDDS-343.003.patch, HDDS-343.004.patch, HDDS-343.005.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be closed.
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 2
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 2 is in CLOSING state and need not be closed.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595307#comment-16595307 ] Elek, Marton commented on HDDS-359: --- +1 Will commit it to the trunk shortly, if no more objections. > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch, > HDDS-359.003.patch, HDDS-359.004.patch, HDDS-359.005.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-280:
--
Attachment: HDDS-280.002.patch
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-280.001.patch, HDDS-280.002.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-369) Remove the containers of a dead node from the container state map
[ https://issues.apache.org/jira/browse/HDDS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-369: -- Attachment: HDDS-369.003.patch > Remove the containers of a dead node from the container state map > - > > Key: HDDS-369 > URL: https://issues.apache.org/jira/browse/HDDS-369 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-369.001.patch, HDDS-369.002.patch, > HDDS-369.003.patch > > > In case of a node is dead we need to update the container replicas > information of the containerStateMap for all the containers from that > specific node. > With removing the replica information we can detect the under replicated > state and start the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-369) Remove the containers of a dead node from the container state map
[
https://issues.apache.org/jira/browse/HDDS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595295#comment-16595295
]
Elek, Marton commented on HDDS-369:
---
Thanks the review [~nandakumar131] and [~ajayydv]. I uploaded a new version
with using ContainerStateManager and parametrized log.
{quote}
Also apart from updating replica info, shouldn't this handler also update
information related to storage (space left/used) for cluster?
{quote}
Good question. It seems that the DEAD_NODE event is sent by the
NodeStateManager. I think the NodeManager part should handle all of the
statistics. But not sure. [~nandakumar131] what do you think?
(But this jira is fixing the container state map in case of dead node. I would
take care the statistics in a follow-up jira, unless it's a very small
modification here)
> Remove the containers of a dead node from the container state map
> -
>
> Key: HDDS-369
> URL: https://issues.apache.org/jira/browse/HDDS-369
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-369.001.patch, HDDS-369.002.patch,
> HDDS-369.003.patch
>
>
> In case of a node is dead we need to update the container replicas
> information of the containerStateMap for all the containers from that
> specific node.
> With removing the replica information we can detect the under replicated
> state and start the replication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-359) RocksDB Profiles support
[ https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-359: -- Resolution: Fixed Status: Resolved (was: Patch Available) Just landed on trunk. Thank you very much [~anu] the contribution. > RocksDB Profiles support > > > Key: HDDS-359 > URL: https://issues.apache.org/jira/browse/HDDS-359 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Xiaoyu Yao >Assignee: Anu Engineer >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-359.001.patch, HDDS-359.002.patch, > HDDS-359.003.patch, HDDS-359.004.patch, HDDS-359.005.patch > > > This allows us to tune the OM/SCM DB for different machine configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-75) Ozone: Support CopyContainer
[ https://issues.apache.org/jira/browse/HDDS-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-75: - Attachment: HDDS-75.011.patch > Ozone: Support CopyContainer > > > Key: HDDS-75 > URL: https://issues.apache.org/jira/browse/HDDS-75 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Anu Engineer >Assignee: Elek, Marton >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-75.005.patch, HDDS-75.006.patch, HDDS-75.007.patch, > HDDS-75.009.patch, HDDS-75.010.patch, HDDS-75.011.patch, > HDFS-11686-HDFS-7240.001.patch, HDFS-11686-HDFS-7240.002.patch, > HDFS-11686-HDFS-7240.003.patch, HDFS-11686-HDFS-7240.004.patch > > > Once a container is closed we need to copy the container to the correct pool > or re-encode the container to use erasure coding. The copyContainer allows > users to get the container as a tarball from the remote machine. > The copyContainer is a basic step to move the raw container data from one > datanode to an other node. It could be used by higher level components such > like the scm which ensures that the replication rules are satisfied. > The CopyContainer by default works in pull model: the destination datanode > could read the raw data from one or more source datanode where the container > exists. > The source provides a binary representation of the container over a common > interface which has two method: > # prepare(containerName) > # copyData(String containerName, OutputStream destination) > Prepare phase is called right after the closing event and the implementation > could prepare for the copy by precreate a compressed tar file from the > container data. As a first step we can provide a simple implementation which > creates the tar files on demand. > The destination datanode should retry the copy if the container in the source > node not yet prepared. > The raw container data is provided over HTTP. The HTTP endpoint should be > separated from the ObjectStore REST API (similar to the distinctions between > HDFS-7240 and HDFS-13074) > Long-term the HTTP endpoint should support Http-Range requests: One container > could be copied from multiple source by the destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-75) Ozone: Support CopyContainer
[ https://issues.apache.org/jira/browse/HDDS-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596063#comment-16596063 ] Elek, Marton commented on HDDS-75: -- The patch is rebased and updated. Checkstyle/findbugs/unit test issues are fixed (hopefully...). > Ozone: Support CopyContainer > > > Key: HDDS-75 > URL: https://issues.apache.org/jira/browse/HDDS-75 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Anu Engineer >Assignee: Elek, Marton >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-75.005.patch, HDDS-75.006.patch, HDDS-75.007.patch, > HDDS-75.009.patch, HDDS-75.010.patch, HDDS-75.011.patch, HDDS-75.012.patch, > HDFS-11686-HDFS-7240.001.patch, HDFS-11686-HDFS-7240.002.patch, > HDFS-11686-HDFS-7240.003.patch, HDFS-11686-HDFS-7240.004.patch > > > Once a container is closed we need to copy the container to the correct pool > or re-encode the container to use erasure coding. The copyContainer allows > users to get the container as a tarball from the remote machine. > The copyContainer is a basic step to move the raw container data from one > datanode to an other node. It could be used by higher level components such > like the scm which ensures that the replication rules are satisfied. > The CopyContainer by default works in pull model: the destination datanode > could read the raw data from one or more source datanode where the container > exists. > The source provides a binary representation of the container over a common > interface which has two method: > # prepare(containerName) > # copyData(String containerName, OutputStream destination) > Prepare phase is called right after the closing event and the implementation > could prepare for the copy by precreate a compressed tar file from the > container data. As a first step we can provide a simple implementation which > creates the tar files on demand. > The destination datanode should retry the copy if the container in the source > node not yet prepared. > The raw container data is provided over HTTP. The HTTP endpoint should be > separated from the ObjectStore REST API (similar to the distinctions between > HDFS-7240 and HDFS-13074) > Long-term the HTTP endpoint should support Http-Range requests: One container > could be copied from multiple source by the destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-75) Ozone: Support CopyContainer
[ https://issues.apache.org/jira/browse/HDDS-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-75: - Attachment: HDDS-75.012.patch > Ozone: Support CopyContainer > > > Key: HDDS-75 > URL: https://issues.apache.org/jira/browse/HDDS-75 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Anu Engineer >Assignee: Elek, Marton >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-75.005.patch, HDDS-75.006.patch, HDDS-75.007.patch, > HDDS-75.009.patch, HDDS-75.010.patch, HDDS-75.011.patch, HDDS-75.012.patch, > HDFS-11686-HDFS-7240.001.patch, HDFS-11686-HDFS-7240.002.patch, > HDFS-11686-HDFS-7240.003.patch, HDFS-11686-HDFS-7240.004.patch > > > Once a container is closed we need to copy the container to the correct pool > or re-encode the container to use erasure coding. The copyContainer allows > users to get the container as a tarball from the remote machine. > The copyContainer is a basic step to move the raw container data from one > datanode to an other node. It could be used by higher level components such > like the scm which ensures that the replication rules are satisfied. > The CopyContainer by default works in pull model: the destination datanode > could read the raw data from one or more source datanode where the container > exists. > The source provides a binary representation of the container over a common > interface which has two method: > # prepare(containerName) > # copyData(String containerName, OutputStream destination) > Prepare phase is called right after the closing event and the implementation > could prepare for the copy by precreate a compressed tar file from the > container data. As a first step we can provide a simple implementation which > creates the tar files on demand. > The destination datanode should retry the copy if the container in the source > node not yet prepared. > The raw container data is provided over HTTP. The HTTP endpoint should be > separated from the ObjectStore REST API (similar to the distinctions between > HDFS-7240 and HDFS-13074) > Long-term the HTTP endpoint should support Http-Range requests: One container > could be copied from multiple source by the destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-75) Ozone: Support CopyContainer
[ https://issues.apache.org/jira/browse/HDDS-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593453#comment-16593453 ] Elek, Marton commented on HDDS-75: -- HDDS-328 is committed. The patch is rebased on top of the trunk. > Ozone: Support CopyContainer > > > Key: HDDS-75 > URL: https://issues.apache.org/jira/browse/HDDS-75 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Anu Engineer >Assignee: Elek, Marton >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-75.005.patch, HDDS-75.006.patch, HDDS-75.007.patch, > HDDS-75.009.patch, HDDS-75.010.patch, HDFS-11686-HDFS-7240.001.patch, > HDFS-11686-HDFS-7240.002.patch, HDFS-11686-HDFS-7240.003.patch, > HDFS-11686-HDFS-7240.004.patch > > > Once a container is closed we need to copy the container to the correct pool > or re-encode the container to use erasure coding. The copyContainer allows > users to get the container as a tarball from the remote machine. > The copyContainer is a basic step to move the raw container data from one > datanode to an other node. It could be used by higher level components such > like the scm which ensures that the replication rules are satisfied. > The CopyContainer by default works in pull model: the destination datanode > could read the raw data from one or more source datanode where the container > exists. > The source provides a binary representation of the container over a common > interface which has two method: > # prepare(containerName) > # copyData(String containerName, OutputStream destination) > Prepare phase is called right after the closing event and the implementation > could prepare for the copy by precreate a compressed tar file from the > container data. As a first step we can provide a simple implementation which > creates the tar files on demand. > The destination datanode should retry the copy if the container in the source > node not yet prepared. > The raw container data is provided over HTTP. The HTTP endpoint should be > separated from the ObjectStore REST API (similar to the distinctions between > HDFS-7240 and HDFS-13074) > Long-term the HTTP endpoint should support Http-Range requests: One container > could be copied from multiple source by the destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-75) Ozone: Support CopyContainer
[ https://issues.apache.org/jira/browse/HDDS-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-75: - Attachment: HDDS-75.010.patch > Ozone: Support CopyContainer > > > Key: HDDS-75 > URL: https://issues.apache.org/jira/browse/HDDS-75 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Anu Engineer >Assignee: Elek, Marton >Priority: Blocker > Fix For: 0.2.1 > > Attachments: HDDS-75.005.patch, HDDS-75.006.patch, HDDS-75.007.patch, > HDDS-75.009.patch, HDDS-75.010.patch, HDFS-11686-HDFS-7240.001.patch, > HDFS-11686-HDFS-7240.002.patch, HDFS-11686-HDFS-7240.003.patch, > HDFS-11686-HDFS-7240.004.patch > > > Once a container is closed we need to copy the container to the correct pool > or re-encode the container to use erasure coding. The copyContainer allows > users to get the container as a tarball from the remote machine. > The copyContainer is a basic step to move the raw container data from one > datanode to an other node. It could be used by higher level components such > like the scm which ensures that the replication rules are satisfied. > The CopyContainer by default works in pull model: the destination datanode > could read the raw data from one or more source datanode where the container > exists. > The source provides a binary representation of the container over a common > interface which has two method: > # prepare(containerName) > # copyData(String containerName, OutputStream destination) > Prepare phase is called right after the closing event and the implementation > could prepare for the copy by precreate a compressed tar file from the > container data. As a first step we can provide a simple implementation which > creates the tar files on demand. > The destination datanode should retry the copy if the container in the source > node not yet prepared. > The raw container data is provided over HTTP. The HTTP endpoint should be > separated from the ObjectStore REST API (similar to the distinctions between > HDFS-7240 and HDFS-13074) > Long-term the HTTP endpoint should support Http-Range requests: One container > could be copied from multiple source by the destination. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-359) RocksDB Profiles support
[
https://issues.apache.org/jira/browse/HDDS-359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593392#comment-16593392
]
Elek, Marton commented on HDDS-359:
---
Thanks Anu the patch. I like this approach especially the flexibility in the
configuration method. As you wrote:
{quote}
Here is what we are doing, if there is a name.db.ini in the config directory,
we will use those params for the RocksDB. Otherwise, we will use the profile
key if specified, if there is nothing specified we will use the default
profile. The return of null makes it easy for us to know that we need to get a
profile from the secondary sources.
{quote}
I have two minor comments (non of them are blockers):
1. Would be great to get some log output about initializing the
table/columnfamiliy with a specific file/profile. Users may not familiar with
this configuration but checking the log it could be a warning if I have only
spinning disk but the log contains something about SSD. Also would be easier to
check if a custom db file is picked. I would be happy even with an INFO level
line.
2. I would use HddsConfigKeys instead of OzoneConfigKeys. The functionality is
part of the core hdds and long term we need to separated the ozone specific and
hdds specific configuration (not a blocker as we have a lot of other
configurations which should be moved to the HddsConfigKeys).
But non of these are blockers, both of them could be improved later. I am +1
even without these...
> RocksDB Profiles support
>
>
> Key: HDDS-359
> URL: https://issues.apache.org/jira/browse/HDDS-359
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Anu Engineer
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-359.001.patch, HDDS-359.002.patch,
> HDDS-359.003.patch, HDDS-359.004.patch
>
>
> This allows us to tune the OM/SCM DB for different machine configurations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-334) Update GettingStarted page to mention details about Ozone GenConf tool
[ https://issues.apache.org/jira/browse/HDDS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-334: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~dineshchitlangia]. It looks good to me. I am just committing it to the trunk. Thank you very much the contribution. > Update GettingStarted page to mention details about Ozone GenConf tool > -- > > Key: HDDS-334 > URL: https://issues.apache.org/jira/browse/HDDS-334 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: document >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Labels: documentation > Fix For: 0.2.1 > > Attachments: HDDS-334.001.patch, HDDS-334.002.patch > > > Add description about Ozone GenConf tool in GettingStarted page -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-334) Update GettingStarted page to mention details about Ozone GenConf tool
[ https://issues.apache.org/jira/browse/HDDS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-334: -- Priority: Minor (was: Major) > Update GettingStarted page to mention details about Ozone GenConf tool > -- > > Key: HDDS-334 > URL: https://issues.apache.org/jira/browse/HDDS-334 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: document >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Minor > Labels: documentation > Fix For: 0.2.1 > > Attachments: HDDS-334.001.patch, HDDS-334.002.patch > > > Add description about Ozone GenConf tool in GettingStarted page -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-379: - Assignee: Elek, Marton > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-379: -- Description: SCMCLI is a useful tool to test SCM. It can create/delete/close/list containers. There are multiple problems with the current scmcli. The biggest one is the cli argument handling. Similar to HDDS-190, it's often very hard to get the help for a specific subcommand. The other one is that a big part of the code is the argument handling which is mixed with the business logic. I propose to use a more modern argument handler library and simplify the argument handling (and improve the user experience). I propose to use [picocli|https://github.com/remkop/picocli]. 1.) It supports subcommands and subcommand specific and general arguments. 2.) It could work based on annotation with very few additional boilerplate code 3.) It's very well documented and easy to use 4.) It's licenced under Apache licence 5.) It supports tab autocompletion for bash and zsh and colorful output 6.) Actively maintainer project 7.) Adopter by other bigger projects (groovy, junit, log4j) In this patch I would like to demonstrate how the cli handling could be simplified. And if it's accepted, we can start to use similar approach for other ozone cli as well. The patch also fixes the cli (the name of the main class was wrong). It also requires HDDS-377 for the be compiled. I also deleted the TestSCMCli. It was turned off with an annotation and I believe that this functionality could be tested more easily with a robot test. was: SCMCLI is a useful tool to test SCM. It can create/delete/close/list containers. There are multiple problems with the current scmcli. The biggest one is the cli argument handling. Similar to HDDS-190, it's often very hard to get the help for a specific subcommand. The other one is that a big part of the code is the argument handling which is mixed with the business logic. I propose to use a more modern argument handler library and simplify the argument handling (and improve the user experience). I propose to use [picocli|https://github.com/remkop/picocli]. 1.) It supports subcommands and subcommand specific and general arguments. 2.) It could work based on annotation with very few additional boilerplate code 3.) It's very well documented and easy to use 4.) It's licenced under Apache licence 5.) It supports tab autocompletion for bash and zsh and colorful output 6.) Actively maintainer project 7.) Adopter by other bigger projects (groovy, junit, log4j) In this patch I would like to demonstrate how the cli handling could be simplified. And if it's accepted, we can start to use similar approach for other ozone cli as well. The patch also fixes the cli (the name of the main class was wrong). It also requires HDDS-377 for the be compiled. > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail:
[jira] [Assigned] (HDDS-377) Make the ScmClient closable and stop the started threads
[
https://issues.apache.org/jira/browse/HDDS-377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton reassigned HDDS-377:
-
Assignee: Elek, Marton
> Make the ScmClient closable and stop the started threads
>
>
> Key: HDDS-377
> URL: https://issues.apache.org/jira/browse/HDDS-377
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM Client
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
>
> Current ScmClient class opens additional threads which won't be closed. For
> example SCMCLI can't stop because of this running thread:
> {code}
> "nioEventLoopGroup-2-1" #15 prio=10 os_prio=0 tid=0x7f1c84c74800
> nid=0x77f4 runnable [0x7f1c52238000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000771adf7b0> (a
> org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000771ae12d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000771ae1010> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
> at
> org.apache.ratis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> at
> org.apache.ratis.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> (Note this is netty, but the Grpc based xceiver also have some special
> threads).
> I propose to make ScmClient auto-closable and stop the XceiverClientManager
> in case of close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-378) Remove dependencies between hdds/ozone and hdfs proto files
Elek, Marton created HDDS-378:
-
Summary: Remove dependencies between hdds/ozone and hdfs proto
files
Key: HDDS-378
URL: https://issues.apache.org/jira/browse/HDDS-378
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Elek, Marton
It would be great to make the hdds/ozone proto files independent from hdfs
proto files. It would help as to start ozone with multiple version of hadoop
version.
Also helps to make artifacts from the hdds protos: HDDS-220
Currently we have a few unused "hdfs.proto" import in the proto files and we
use the StorageTypeProto from hdfs:
{code}
cd hadoop-hdds
grep -r "hdfs" --include="*.proto"
common/src/main/proto/ScmBlockLocationProtocol.proto:import "hdfs.proto";
common/src/main/proto/StorageContainerLocationProtocol.proto:import
"hdfs.proto";
cd ../hadoop-ozone
grep -r "hdfs" --include="*.proto"
common/src/main/proto/OzoneManagerProtocol.proto:import "hdfs.proto";
common/src/main/proto/OzoneManagerProtocol.proto:required
hadoop.hdfs.StorageTypeProto storageType = 5 [default = DISK];
common/src/main/proto/OzoneManagerProtocol.proto:optional
hadoop.hdfs.StorageTypeProto storageType = 6;
{code}
I propose to
1.) remove the hdfs import statements from the proto files
2.) Copy the StorageTypeProto and create a Hdds version from it (without
PROVIDED)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-190) Improve shell error message for unrecognized option
[
https://issues.apache.org/jira/browse/HDDS-190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593589#comment-16593589
]
Elek, Marton commented on HDDS-190:
---
Hi [~Sandeep Nemuri]. Are you working on this? I have a proposal to solve this
issue...
> Improve shell error message for unrecognized option
> ---
>
> Key: HDDS-190
> URL: https://issues.apache.org/jira/browse/HDDS-190
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Sandeep Nemuri
>Priority: Major
> Labels: newbie
> Fix For: 0.2.1
>
>
> The error message with an unrecognized option is unfriendly. E.g.
> {code}
> $ ozone oz -badOption
> Unrecognized option: -badOptionERROR: null
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-379: -- Attachment: HDDS-379.001.patch > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-379.001.patch > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-379: -- Status: Patch Available (was: Open) > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-379.001.patch > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-377) Make the ScmClient closable and stop the started threads
Elek, Marton created HDDS-377:
-
Summary: Make the ScmClient closable and stop the started threads
Key: HDDS-377
URL: https://issues.apache.org/jira/browse/HDDS-377
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: SCM Client
Reporter: Elek, Marton
Fix For: 0.2.1
Current ScmClient class opens additional threads which won't be closed. For
example SCMCLI can't stop because of this running thread:
{code}
"nioEventLoopGroup-2-1" #15 prio=10 os_prio=0 tid=0x7f1c84c74800 nid=0x77f4
runnable [0x7f1c52238000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000771adf7b0> (a
org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x000771ae12d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000771ae1010> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at
org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
at
org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
at
org.apache.ratis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at
org.apache.ratis.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:745)
{code}
(Note this is netty, but the Grpc based xceiver also have some special threads).
I propose to make ScmClient auto-closable and stop the XceiverClientManager in
case of close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-377) Make the ScmClient closable and stop the started threads
[
https://issues.apache.org/jira/browse/HDDS-377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-377:
--
Attachment: HDDS-377.001.patch
> Make the ScmClient closable and stop the started threads
>
>
> Key: HDDS-377
> URL: https://issues.apache.org/jira/browse/HDDS-377
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM Client
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-377.001.patch
>
>
> Current ScmClient class opens additional threads which won't be closed. For
> example SCMCLI can't stop because of this running thread:
> {code}
> "nioEventLoopGroup-2-1" #15 prio=10 os_prio=0 tid=0x7f1c84c74800
> nid=0x77f4 runnable [0x7f1c52238000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000771adf7b0> (a
> org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000771ae12d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000771ae1010> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
> at
> org.apache.ratis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> at
> org.apache.ratis.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> (Note this is netty, but the Grpc based xceiver also have some special
> threads).
> I propose to make ScmClient auto-closable and stop the XceiverClientManager
> in case of close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-377) Make the ScmClient closable and stop the started threads
[
https://issues.apache.org/jira/browse/HDDS-377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-377:
--
Status: Patch Available (was: Open)
> Make the ScmClient closable and stop the started threads
>
>
> Key: HDDS-377
> URL: https://issues.apache.org/jira/browse/HDDS-377
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM Client
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-377.001.patch
>
>
> Current ScmClient class opens additional threads which won't be closed. For
> example SCMCLI can't stop because of this running thread:
> {code}
> "nioEventLoopGroup-2-1" #15 prio=10 os_prio=0 tid=0x7f1c84c74800
> nid=0x77f4 runnable [0x7f1c52238000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x000771adf7b0> (a
> org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySet)
> - locked <0x000771ae12d8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x000771ae1010> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753)
> at
> org.apache.ratis.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409)
> at
> org.apache.ratis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> at
> org.apache.ratis.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> (Note this is netty, but the Grpc based xceiver also have some special
> threads).
> I propose to make ScmClient auto-closable and stop the XceiverClientManager
> in case of close.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
Elek, Marton created HDDS-379: - Summary: Simplify and improve the cli arg parsing of ozone scmcli Key: HDDS-379 URL: https://issues.apache.org/jira/browse/HDDS-379 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Elek, Marton Fix For: 0.2.1 SCMCLI is a useful tool to test SCM. It can create/delete/close/list containers. There are multiple problems with the current scmcli. The biggest one is the cli argument handling. Similar to HDDS-190, it's often very hard to get the help for a specific subcommand. The other one is that a big part of the code is the argument handling which is mixed with the business logic. I propose to use a more modern argument handler library and simplify the argument handling (and improve the user experience). I propose to use [picocli|https://github.com/remkop/picocli]. 1.) It supports subcommands and subcommand specific and general arguments. 2.) It could work based on annotation with very few additional boilerplate code 3.) It's very well documented and easy to use 4.) It's licenced under Apache licence 5.) It supports tab autocompletion for bash and zsh and colorful output 6.) Actively maintainer project 7.) Adopter by other bigger projects (groovy, junit, log4j) In this patch I would like to demonstrate how the cli handling could be simplified. And if it's accepted, we can start to use similar approach for other ozone cli as well. The patch also fixes the cli (the name of the main class was wrong). It also requires HDDS-377 for the be compiled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-313) Add metrics to containerState Machine
[ https://issues.apache.org/jira/browse/HDDS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-313: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to the trunk. Thank you the contribution. > Add metrics to containerState Machine > - > > Key: HDDS-313 > URL: https://issues.apache.org/jira/browse/HDDS-313 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-313.001.patch, HDDS-313.002.patch, > HDDS-313.003.patch, HDDS-313.004.patch, HDDS-313.005.patch, > HDDS-313.006.patch, HDDS-313.007.patch > > > metrics needs to be added to containerStateMachine to keep track of various > ratis ops like writeStateMachine/readStateMachine/applyTransactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-328) Support export and import of the KeyValueContainer
[ https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-328: -- Attachment: HDDS-328.009.patch > Support export and import of the KeyValueContainer > -- > > Key: HDDS-328 > URL: https://issues.apache.org/jira/browse/HDDS-328 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Critical > Fix For: 0.2.1 > > Attachments: HDDS-328-HDFS-10285.006.patch, HDDS-328.002.patch, > HDDS-328.003.patch, HDDS-328.004.patch, HDDS-328.005.patch, > HDDS-328.007.patch, HDDS-328.008.patch, HDDS-328.009.patch > > > In HDDS-75 we pack the container data to an archive file, copy to other > datanodes and create the container from the archive. > As I wrote in the comment of HDDS-75 I propose to separate the patch to make > it easier to review. > In this patch we need to extend the existing Container interface with adding > export/import methods to save the container data to one binary input/output > stream. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-227) Use Grpc as the default transport protocol for Standalone pipeline
[ https://issues.apache.org/jira/browse/HDDS-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-227: - Assignee: chencan > Use Grpc as the default transport protocol for Standalone pipeline > -- > > Key: HDDS-227 > URL: https://issues.apache.org/jira/browse/HDDS-227 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Attachments: HDDS-227.001.patch > > > Using a config, Standalone pipeline can currently choose between Grpc and > Netty based transport protocol, this jira proposes to use only grpc as the > transport protocol. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-364) Update open container replica information in SCM during DN register
[ https://issues.apache.org/jira/browse/HDDS-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588633#comment-16588633 ] Elek, Marton commented on HDDS-364: --- Thank you very much [~ajayydv] to work on this. Overall it looks good to me and it was definitely missing until now. 1. What do you think about Node2ContainerMap? Do we need to update it as well? 2. As I see all the containers will be persisted twice. (First, they will be imported, after that they will be reconciled.). Don't think it's a big problem. IMHO later we need to cleanup all the processing path anyway. One option may be just saving all the initial data to the state map without processing the reports (checking the required closing state, etc.). The downside here is some action would be delayed until the first real container report. 3. The import part (in case of isRegisterCall=true) is the first part of processContainerReport method. I think it would be very easy to move to a separated method and call it independently from SCMDatanodeProtocolServer.register method. Could be more simple, and maybe it could be easier to test. Currently (as I understood) there is no specific test to test the isRegisterCall=true path. But this is not a blocking problem. Depends from your consideration... > Update open container replica information in SCM during DN register > --- > > Key: HDDS-364 > URL: https://issues.apache.org/jira/browse/HDDS-364 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-364.00.patch, HDDS-364.01.patch > > > Update open container replica information in SCM during DN register. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-227) Use Grpc as the default transport protocol for Standalone pipeline
[ https://issues.apache.org/jira/browse/HDDS-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588650#comment-16588650 ] Elek, Marton commented on HDDS-227: --- +1. Waiting for the Jenkins but AFAIK this is exactly what we need. Thank you very much [~candychencan] for the patch. (Yesterday I just lost a few hours because I didn't switch to grpc in the right way. So it definitely a big help for HDDS-75). > Use Grpc as the default transport protocol for Standalone pipeline > -- > > Key: HDDS-227 > URL: https://issues.apache.org/jira/browse/HDDS-227 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Attachments: HDDS-227.001.patch > > > Using a config, Standalone pipeline can currently choose between Grpc and > Netty based transport protocol, this jira proposes to use only grpc as the > transport protocol. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-363) Faster datanode registration during the first startup
[ https://issues.apache.org/jira/browse/HDDS-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-363: -- Status: Patch Available (was: Open) > Faster datanode registration during the first startup > - > > Key: HDDS-363 > URL: https://issues.apache.org/jira/browse/HDDS-363 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-363.001.patch > > > During the first startup usually we need to wait about 30 s to find the scm > usable. The datanode registration is a multiple step process > (request/response + request/response) and we need to wait the next HB to > finish the registration. > I propose to use a more higher HB frequency at startup (let's say 2 seconds) > and set the configured HB only at the end of the registration. > It also helps for the first users as it could be less confusing (the datanode > can be seen almost immediately on the UI) > Also it would help a lot for me during the testing (yes, I can decrease the > HB frequency but in that case it's harder the follow the later HBs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-363) Faster datanode registration during the first startup
[ https://issues.apache.org/jira/browse/HDDS-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-363: -- Attachment: HDDS-363.001.patch > Faster datanode registration during the first startup > - > > Key: HDDS-363 > URL: https://issues.apache.org/jira/browse/HDDS-363 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-363.001.patch > > > During the first startup usually we need to wait about 30 s to find the scm > usable. The datanode registration is a multiple step process > (request/response + request/response) and we need to wait the next HB to > finish the registration. > I propose to use a more higher HB frequency at startup (let's say 2 seconds) > and set the configured HB only at the end of the registration. > It also helps for the first users as it could be less confusing (the datanode > can be seen almost immediately on the UI) > Also it would help a lot for me during the testing (yes, I can decrease the > HB frequency but in that case it's harder the follow the later HBs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-369) Remove the containers of a dead node from the container state map
[ https://issues.apache.org/jira/browse/HDDS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-369: -- Attachment: HDDS-369.001.patch > Remove the containers of a dead node from the container state map > - > > Key: HDDS-369 > URL: https://issues.apache.org/jira/browse/HDDS-369 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-369.001.patch > > > In case of a node is dead we need to update the container replicas > information of the containerStateMap for all the containers from that > specific node. > With removing the replica information we can detect the under replicated > state and start the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-369) Remove the containers of a dead node from the container state map
[ https://issues.apache.org/jira/browse/HDDS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton reassigned HDDS-369: - Assignee: Elek, Marton > Remove the containers of a dead node from the container state map > - > > Key: HDDS-369 > URL: https://issues.apache.org/jira/browse/HDDS-369 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-369.001.patch > > > In case of a node is dead we need to update the container replicas > information of the containerStateMap for all the containers from that > specific node. > With removing the replica information we can detect the under replicated > state and start the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-369) Remove the containers of a dead node from the container state map
[ https://issues.apache.org/jira/browse/HDDS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-369: -- Status: Patch Available (was: Open) > Remove the containers of a dead node from the container state map > - > > Key: HDDS-369 > URL: https://issues.apache.org/jira/browse/HDDS-369 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-369.001.patch > > > In case of a node is dead we need to update the container replicas > information of the containerStateMap for all the containers from that > specific node. > With removing the replica information we can detect the under replicated > state and start the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-363) Faster datanode registration during the first startup
[
https://issues.apache.org/jira/browse/HDDS-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589112#comment-16589112
]
Elek, Marton commented on HDDS-363:
---
Acceptance tests are passed:
{code}
==
Acceptance
==
Acceptance.Basic
==
Acceptance.Basic.Basic :: Smoketest ozone cluster startup
==
Test rest interface | PASS |
--
Check webui static resources | PASS |
--
Start freon testing | PASS |
--
Acceptance.Basic.Basic :: Smoketest ozone cluster startup | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==
Acceptance.Basic.Ozone-Shell :: Test ozone shell CLI usage
==
RestClient without http port | PASS |
--
RestClient with http port | PASS |
--
RestClient without host name | PASS |
--
RpcClient with port | PASS |
--
RpcClient without host| PASS |
--
RpcClient without scheme | PASS |
--
Acceptance.Basic.Ozone-Shell :: Test ozone shell CLI usage| PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==
Acceptance.Basic | PASS |
9 critical tests, 9 passed, 0 failed
9 tests total, 9 passed, 0 failed
==
Acceptance.Ozonefs
==
Acceptance.Ozonefs.Ozonefs :: Ozonefs test
==
Create volume and bucket | PASS |
--
Check volume from ozonefs | PASS |
--
Create directory from ozonefs | PASS |
--
Acceptance.Ozonefs.Ozonefs :: Ozonefs test| PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==
Acceptance.Ozonefs.Ozonesinglenode :: Ozonefs Single Node Test
==
Create volume and bucket | PASS |
--
Check volume from ozonefs | PASS |
--
Create directory from ozonefs | PASS |
--
Test key handling | PASS |
--
Acceptance.Ozonefs.Ozonesinglenode :: Ozonefs Single Node Test| PASS |
4 critical tests, 4
[jira] [Created] (HDDS-369) Remove the containers of a dead node from the container state map
Elek, Marton created HDDS-369: - Summary: Remove the containers of a dead node from the container state map Key: HDDS-369 URL: https://issues.apache.org/jira/browse/HDDS-369 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Elek, Marton Fix For: 0.2.1 In case of a node is dead we need to update the container replicas information of the containerStateMap for all the containers from that specific node. With removing the replica information we can detect the under replicated state and start the replication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-325) Add event watcher for delete blocks command
[
https://issues.apache.org/jira/browse/HDDS-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589109#comment-16589109
]
Elek, Marton commented on HDDS-325:
---
Thanks [~ljain] the update. Unfortunately he patch doesn't apply any more:
{code}
Attachments:
Downloading HDDS-325.004.patch
exit status 1
error: patch failed:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java:67
error:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java:
patch does not apply
error: patch failed:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestBlockDeletion.java:60
error:
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestBlockDeletion.java:
patch does not apply
{code}
Would you be so kind to rebase it?
> Add event watcher for delete blocks command
> ---
>
> Key: HDDS-325
> URL: https://issues.apache.org/jira/browse/HDDS-325
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode, SCM
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-325.001.patch, HDDS-325.002.patch,
> HDDS-325.003.patch, HDDS-325.004.patch
>
>
> This Jira aims to add watcher for deleteBlocks command. It removes the
> current rpc call required for datanode to send the acknowledgement for
> deleteBlocks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-363) Faster datanode registration during the first startup
[ https://issues.apache.org/jira/browse/HDDS-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-363: -- Attachment: HDDS-363.002.patch > Faster datanode registration during the first startup > - > > Key: HDDS-363 > URL: https://issues.apache.org/jira/browse/HDDS-363 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-363.001.patch, HDDS-363.002.patch > > > During the first startup usually we need to wait about 30 s to find the scm > usable. The datanode registration is a multiple step process > (request/response + request/response) and we need to wait the next HB to > finish the registration. > I propose to use a more higher HB frequency at startup (let's say 2 seconds) > and set the configured HB only at the end of the registration. > It also helps for the first users as it could be less confusing (the datanode > can be seen almost immediately on the UI) > Also it would help a lot for me during the testing (yes, I can decrease the > HB frequency but in that case it's harder the follow the later HBs) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-328) Support export and import of the KeyValueContainer
[ https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587606#comment-16587606 ] Elek, Marton commented on HDDS-328: --- Working on the last remaining issue. Will be fixed soon > Support export and import of the KeyValueContainer > -- > > Key: HDDS-328 > URL: https://issues.apache.org/jira/browse/HDDS-328 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Critical > Fix For: 0.2.1 > > Attachments: HDDS-328-HDFS-10285.006.patch, HDDS-328.002.patch, > HDDS-328.003.patch, HDDS-328.004.patch, HDDS-328.005.patch > > > In HDDS-75 we pack the container data to an archive file, copy to other > datanodes and create the container from the archive. > As I wrote in the comment of HDDS-75 I propose to separate the patch to make > it easier to review. > In this patch we need to extend the existing Container interface with adding > export/import methods to save the container data to one binary input/output > stream. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-336) Print out container location information for a specific ozone key
[
https://issues.apache.org/jira/browse/HDDS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591475#comment-16591475
]
Elek, Marton commented on HDDS-336:
---
Thank you very much the patch [~GeLiXin]. Overall it looks very good to me.
I have only two concerns:
1. One container could have multiple blocks and one block is defined with
containerid + localid pair. Because ContainerId is not unique the
Map is not good enough to store the location information. (one
container could have (c1,l1), (c1,l2) blocks an in that case the map would do a
c1->l2 from this information)
It could be Map.Entry but creating a specific class would be even
better, because it would be more clean what are meanings of the fields.
For example:
{code}
public class OzoneKeyLocation {
private final long containerId;
private final long localId;
private final long length;
private final long offset;
{code}
And List> could be changed to List.
(length and offsets are not hard requirement buts with getting this information
we can check if we have all the bytes for a specific key )
2. this could be finished in a separated jira, but I would add it to here as
this is the best part: it would be great to print out these information to the
console using the 'ozone oz' cli. As you modified all the required code part,
this is the easy endgame: ClientProtocol.getKeyDetails and OzoneBucket.getKey
could be modified to return with OzoneKeyDetails. And at the end of the
InfoKeyHandler we can print out the containerid->localid mapping.
it could be tested with the docker pseudo cluster:
1. do a full build
2. do a 'docker-compose up -d' in the hadoop-dist/target/ozone/compose/ozone
directory
3. do a 'docker-compose scale datanode=3' (just to have 3 datanodes)
4. wait until you can see 3 HEALTHY nodes on localhost:9876 (host could be
different for mac/windows)
5. use 'docker-compose exec datanode bash'
6. Try out the ozone oz cli:
{code}
ozone oz -createVolume /vol1 -user hadoop --quota 1TB --root
ozone oz -createBucket /vol1/bucket
dd if=/dev/zero of=/tmp/test bs=1024000 count=512
ozone oz -putKey /vol1/bucket/file1 -replicationFactor 3 -file /tmp/test
ozone oz -infoKey /vol1/bucket/file1
{code}
Thank you again the patch, this is a very precious implementation of the
original plan.
> Print out container location information for a specific ozone key
> --
>
> Key: HDDS-336
> URL: https://issues.apache.org/jira/browse/HDDS-336
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Assignee: LiXin Ge
>Priority: Major
> Labels: newbie
> Fix For: 0.2.1
>
> Attachments: HDDS-336.000.patch, HDDS-336.001.patch,
> HDDS-336.002.patch
>
>
> In the protobuf protocol we have all the containerid/localid(=blockid)
> information for a specific ozone key.
> It would be a big help to print out this information to the command line with
> the ozone cli.
> It requires to improve the REST and RPC interface with additionalOzone
> KeyLocation information.
> It would help a very big help during the test of the current scm behaviour.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-313) Add metrics to containerState Machine
[ https://issues.apache.org/jira/browse/HDDS-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593678#comment-16593678 ] Elek, Marton commented on HDDS-313: --- Thanks [~candychencan]. Tested with a docker-compose based pseudo cluster and worked well. Can see the metrics over the jmx interface after put key. Unit test is not related. +1. Will commit it to the trunk, soon. > Add metrics to containerState Machine > - > > Key: HDDS-313 > URL: https://issues.apache.org/jira/browse/HDDS-313 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-313.001.patch, HDDS-313.002.patch, > HDDS-313.003.patch, HDDS-313.004.patch, HDDS-313.005.patch, > HDDS-313.006.patch, HDDS-313.007.patch > > > metrics needs to be added to containerStateMachine to keep track of various > ratis ops like writeStateMachine/readStateMachine/applyTransactions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-227) Use Grpc as the default transport protocol for Standalone pipeline
[ https://issues.apache.org/jira/browse/HDDS-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-227: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to the trunk. Thank you very much the contribution. > Use Grpc as the default transport protocol for Standalone pipeline > -- > > Key: HDDS-227 > URL: https://issues.apache.org/jira/browse/HDDS-227 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Mukul Kumar Singh >Assignee: chencan >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-227.001.patch, HDDS-227.002.patch, > HDDS-227.003.patch, HDDS-227.004.patch, HDDS-227.005.patch > > > Using a config, Standalone pipeline can currently choose between Grpc and > Netty based transport protocol, this jira proposes to use only grpc as the > transport protocol. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-227) Use Grpc as the default transport protocol for Standalone pipeline
[
https://issues.apache.org/jira/browse/HDDS-227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593720#comment-16593720
]
Elek, Marton commented on HDDS-227:
---
Thanks [~candychencan]. It looks good to me. I ran a full acceptance test and
everything worked well:
{code}
+++ dirname
/home/elek/projects/hadoop/hadoop-ozone/acceptance-test/dev-support/bin/robot.sh
++ cd /home/elek/projects/hadoop/hadoop-ozone/acceptance-test/dev-support/bin
++ pwd
+ DIR=/home/elek/projects/hadoop/hadoop-ozone/acceptance-test/dev-support/bin
++ which robot
+ '[' '!' /home/elek/.pyenv/shims/robot ']'
+
OZONEDISTDIR=/home/elek/projects/hadoop/hadoop-ozone/acceptance-test/dev-support/bin/../../../../hadoop-dist/target/ozone
+ '[' '!' -d
/home/elek/projects/hadoop/hadoop-ozone/acceptance-test/dev-support/bin/../../../../hadoop-dist/target/ozone
']'
+ robot -x junit-results.xml
/home/elek/projects/hadoop/hadoop-ozone/acceptance-test/dev-support/bin/../../src/test/acceptance
==
Acceptance
==
Acceptance.Basic
==
Acceptance.Basic.Basic :: Smoketest ozone cluster startup
==
Test rest interface | PASS |
--
Check webui static resources | PASS |
--
Start freon testing | PASS |
--
Acceptance.Basic.Basic :: Smoketest ozone cluster startup | PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==
Acceptance.Basic.Ozone-Shell :: Test ozone shell CLI usage
==
RestClient without http port | PASS |
--
RestClient with http port | PASS |
--
RestClient without host name | PASS |
--
RpcClient with port | PASS |
--
RpcClient without host| PASS |
--
RpcClient without scheme | PASS |
--
Acceptance.Basic.Ozone-Shell :: Test ozone shell CLI usage| PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==
Acceptance.Basic | PASS |
9 critical tests, 9 passed, 0 failed
9 tests total, 9 passed, 0 failed
==
Acceptance.Ozonefs
==
Acceptance.Ozonefs.Ozonefs :: Ozonefs test
==
Create volume and bucket | PASS |
--
Check volume from ozonefs | PASS |
--
Create directory from ozonefs | PASS |
--
Acceptance.Ozonefs.Ozonefs :: Ozonefs test| PASS |
3 critical tests, 3 passed, 0 failed
3 tests total, 3 passed, 0 failed
==
Acceptance.Ozonefs.Ozonesinglenode :: Ozonefs Single Node Test
[jira] [Commented] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596284#comment-16596284 ] Elek, Marton commented on HDDS-379: --- v002 addresses the findbug issue. > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-379.001.patch, HDDS-379.002.patch > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-379) Simplify and improve the cli arg parsing of ozone scmcli
[ https://issues.apache.org/jira/browse/HDDS-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-379: -- Attachment: HDDS-379.002.patch > Simplify and improve the cli arg parsing of ozone scmcli > > > Key: HDDS-379 > URL: https://issues.apache.org/jira/browse/HDDS-379 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-379.001.patch, HDDS-379.002.patch > > > SCMCLI is a useful tool to test SCM. It can create/delete/close/list > containers. > There are multiple problems with the current scmcli. > The biggest one is the cli argument handling. Similar to HDDS-190, it's often > very hard to get the help for a specific subcommand. > The other one is that a big part of the code is the argument handling which > is mixed with the business logic. > I propose to use a more modern argument handler library and simplify the > argument handling (and improve the user experience). > I propose to use [picocli|https://github.com/remkop/picocli]. > 1.) It supports subcommands and subcommand specific and general arguments. > 2.) It could work based on annotation with very few additional boilerplate > code > 3.) It's very well documented and easy to use > 4.) It's licenced under Apache licence > 5.) It supports tab autocompletion for bash and zsh and colorful output > 6.) Actively maintainer project > 7.) Adopter by other bigger projects (groovy, junit, log4j) > In this patch I would like to demonstrate how the cli handling could be > simplified. And if it's accepted, we can start to use similar approach for > other ozone cli as well. > The patch also fixes the cli (the name of the main class was wrong). > It also requires HDDS-377 for the be compiled. > I also deleted the TestSCMCli. It was turned off with an annotation and I > believe that this functionality could be tested more easily with a robot test. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-280:
--
Attachment: HDDS-280.003.patch
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-280.001.patch, HDDS-280.002.patch,
> HDDS-280.003.patch
>
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-222) Remove hdfs command line from ozone distrubution.
[ https://issues.apache.org/jira/browse/HDDS-222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-222: -- Status: In Progress (was: Patch Available) Thanks the feedback. Moving back to in-progress, trying to reproduce the problem with integration tests. (Maybe some OSX/linux differences?) > Remove hdfs command line from ozone distrubution. > - > > Key: HDDS-222 > URL: https://issues.apache.org/jira/browse/HDDS-222 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie > Fix For: 0.2.1 > > Attachments: HDDS-222.001.patch > > > As the ozone release artifact doesn't contain a stable namenode/datanode code > the hdfs command should be removed from the ozone artifact. > ozone-dist-layout-stitching also could be simplified to copy only the > required jar files (we don't need to copy the namenode/datanode server side > jars, just the common artifacts -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-276) Fix symbolic link creation during Ozone dist process
[ https://issues.apache.org/jira/browse/HDDS-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-276: -- Attachment: HDDS-276.001.patch > Fix symbolic link creation during Ozone dist process > > > Key: HDDS-276 > URL: https://issues.apache.org/jira/browse/HDDS-276 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.2.1 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Attachments: HDDS-276.001.patch > > > Ozone is creating a symlink during the dist process. > Using the "ozone" directory as a destination name all the docker-based > acceptance tests and docker-compose files are more simple as they don't need > to have the version information in the path. > But to keep the version specific folder name in the tar file we create a > symbolic link during the tar creation. With the symbolic link and the > '–dereference' tar argument we can create the tar file which includes a > versioned directory (ozone-0.2.1) but we can use the a dist directory without > the version in the name (hadoop-dist/target/ozone). > Currently this symlink creation has an issue. It couldn't be run twice. You > need to do a 'mvn clean' before you can create a new dist. > But fortunately this could be fixed easily by checking if the destination > symlink exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-199) Implement ReplicationManager to handle underreplication of closed containers
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-199: -- Attachment: HDDS-199.012.patch > Implement ReplicationManager to handle underreplication of closed containers > > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-199.001.patch, HDDS-199.002.patch, > HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch, > HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch, > HDDS-199.009.patch, HDDS-199.010.patch, HDDS-199.011.patch, HDDS-199.012.patch > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-199) Implement ReplicationManager to handle underreplication of closed containers
[
https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550729#comment-16550729
]
Elek, Marton commented on HDDS-199:
---
Thanks [~xyao] the review:
{quote}SCMEvents.java
Line 34-35: NIT: unused imports
SCMContainerPlacementRandom.java
Line 92-93: NIT: blank change.
SCMContainerPlacementCapacity.java
Line 100: can be removed as the super.chooseDatanodes() already removed the
excludedNodes?
StorageContainerManager.java
Line 222: we need to ensure the LeaseManager instance
commandWatcherLeaseManager is shutdown upon SCM stop around line 585.
ReplicationCommandWatcher.java
Line 36: NIT: unused imports
TestReplicationManager.java
Line 139: should we put it within try{} final{} to ensure the proper stop of
lease manger?
{quote}
Thanks the reporting. They are fixed in the latest patch. LeaseManger is
closed, try/finaly is added, etc.
{quote}ReplicationManager.java
Line 160: please update the title of the JIRA to reflect we handle under
replicated container only after this and open a separate Jira.
{quote}
You are right, HDDS-274 is opened.
{quote}ScmConfigKeys.java
Line 250: Update TestCommonConfigurationFields?
{quote}
Not sure what is required. As I know If I added the key to the
ozone-default.xml the test should be fine. I found that the new key of
HddsConfigKeys.class (after HDDS-187) is missing from the
TestOzoneConfigurationFields. I fixed it in the latest patch but it also could
be commited as a separated fix. (Not a big change so I have no problem to add
it to this patch.
> Implement ReplicationManager to handle underreplication of closed containers
>
>
> Key: HDDS-199
> URL: https://issues.apache.org/jira/browse/HDDS-199
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-199.001.patch, HDDS-199.002.patch,
> HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch,
> HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch,
> HDDS-199.009.patch, HDDS-199.010.patch, HDDS-199.011.patch, HDDS-199.012.patch
>
>
> HDDS/Ozone supports Open and Closed containers. In case of specific
> conditions (container is full, node is failed) the container will be closed
> and will be replicated in a different way. The replication of Open containers
> are handled with Ratis and PipelineManger.
> The ReplicationManager should handle the replication of the ClosedContainers.
> The replication information will be sent as an event
> (UnderReplicated/OverReplicated).
> The Replication manager will collect all of the events in a priority queue
> (to replicate first the containers where more replica is missing) calculate
> the destination datanode (first with a very simple algorithm, later with
> calculating scatter-width) and send the Copy/Delete container to the datanode
> (CommandQueue).
> A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the
> copy/delete in case of failure. This is an in-memory structure (based on
> HDDS-195) which can requeue the underreplicated/overreplicated events to the
> prioirity queue unless the confirmation of the copy/delete command is arrived.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-276) Fix symbolic link creation during Ozone dist process
[ https://issues.apache.org/jira/browse/HDDS-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550757#comment-16550757 ] Elek, Marton commented on HDDS-276: --- [~nandakumar131]: As I remember you also had a symlink related issue. Please let me know if you have the problem after this patch. > Fix symbolic link creation during Ozone dist process > > > Key: HDDS-276 > URL: https://issues.apache.org/jira/browse/HDDS-276 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.2.1 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Attachments: HDDS-276.001.patch > > > Ozone is creating a symlink during the dist process. > Using the "ozone" directory as a destination name all the docker-based > acceptance tests and docker-compose files are more simple as they don't need > to have the version information in the path. > But to keep the version specific folder name in the tar file we create a > symbolic link during the tar creation. With the symbolic link and the > '–dereference' tar argument we can create the tar file which includes a > versioned directory (ozone-0.2.1) but we can use the a dist directory without > the version in the name (hadoop-dist/target/ozone). > Currently this symlink creation has an issue. It couldn't be run twice. You > need to do a 'mvn clean' before you can create a new dist. > But fortunately this could be fixed easily by checking if the destination > symlink exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-274) Handle overreplication in ReplicationManager
Elek, Marton created HDDS-274: - Summary: Handle overreplication in ReplicationManager Key: HDDS-274 URL: https://issues.apache.org/jira/browse/HDDS-274 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: SCM Reporter: Elek, Marton Assignee: Elek, Marton Fix For: 0.2.1 HDDS-199 provides the framework to handle over/under replicated containers, but it contains implementation only for the under replicated containers. The over replicated containers should be handled and should be deleted from the datanodes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-199) Implement ReplicationManager to handle underreplication of closed containers
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-199: -- Summary: Implement ReplicationManager to handle underreplication of closed containers (was: Implement ReplicationManager to replicate Closed Containers) > Implement ReplicationManager to handle underreplication of closed containers > > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-199.001.patch, HDDS-199.002.patch, > HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch, > HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch, > HDDS-199.009.patch, HDDS-199.010.patch, HDDS-199.011.patch > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-276) Fix symbolic link creation during Ozone dist process
Elek, Marton created HDDS-276: - Summary: Fix symbolic link creation during Ozone dist process Key: HDDS-276 URL: https://issues.apache.org/jira/browse/HDDS-276 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.2.1 Reporter: Elek, Marton Assignee: Elek, Marton Ozone is creating a symlink during the dist process. Using the "ozone" directory as a destination name all the docker-based acceptance tests and docker-compose files are more simple as they don't need to have the version information in the path. But to keep the version specific folder name in the tar file we create a symbolic link during the tar creation. With the symbolic link and the '–dereference' tar argument we can create the tar file which includes a versioned directory (ozone-0.2.1) but we can use the a dist directory without the version in the name (hadoop-dist/target/ozone). Currently this symlink creation has an issue. It couldn't be run twice. You need to do a 'mvn clean' before you can create a new dist. But fortunately this could be fixed easily by checking if the destination symlink exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-276) Fix symbolic link creation during Ozone dist process
[ https://issues.apache.org/jira/browse/HDDS-276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-276: -- Status: Patch Available (was: Open) > Fix symbolic link creation during Ozone dist process > > > Key: HDDS-276 > URL: https://issues.apache.org/jira/browse/HDDS-276 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.2.1 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Attachments: HDDS-276.001.patch > > > Ozone is creating a symlink during the dist process. > Using the "ozone" directory as a destination name all the docker-based > acceptance tests and docker-compose files are more simple as they don't need > to have the version information in the path. > But to keep the version specific folder name in the tar file we create a > symbolic link during the tar creation. With the symbolic link and the > '–dereference' tar argument we can create the tar file which includes a > versioned directory (ozone-0.2.1) but we can use the a dist directory without > the version in the name (hadoop-dist/target/ozone). > Currently this symlink creation has an issue. It couldn't be run twice. You > need to do a 'mvn clean' before you can create a new dist. > But fortunately this could be fixed easily by checking if the destination > symlink exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-199) Implement ReplicationManager to handle underreplication of closed containers
[
https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550734#comment-16550734
]
Elek, Marton commented on HDDS-199:
---
{quote}
Shall we rename {{SCMEvents.REPLICATION_COMPLETE}} to something like
{{REPLICATION_STATUS}}. Also we might have to override default logic of
EventWatcher as this replication status object might have 3 possible status
(i.e PENDING, EXECUTED, FAILED). Default method in EventWatcher only handles
EXECUTED.
{quote}
Good question [~ajayydv]. I would address it in a different jira as this patch
became just bigger and bigger.
# Currently the EventWatcher is listening on one completion event not
filtering status event.
# It could be modified but in that case I would also rename the
EventWatcher.completionEvent field and maybe modify the structure (this is the
reason why I prefer to do it in a separated task, It's more like an
EventWatcher adjustment)
# An other option is to modify the SCMDatanodeHeartbeatDispatcher to send
different internal events based on the status of the CommandStatusReport. One
big advantage of this approach that the different type of results (failed
closing, executed closing) will be visible on the EventQueue monitoring
interface (eg. number of failed closing events instead of number of command
status report)
> Implement ReplicationManager to handle underreplication of closed containers
>
>
> Key: HDDS-199
> URL: https://issues.apache.org/jira/browse/HDDS-199
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-199.001.patch, HDDS-199.002.patch,
> HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch,
> HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch,
> HDDS-199.009.patch, HDDS-199.010.patch, HDDS-199.011.patch, HDDS-199.012.patch
>
>
> HDDS/Ozone supports Open and Closed containers. In case of specific
> conditions (container is full, node is failed) the container will be closed
> and will be replicated in a different way. The replication of Open containers
> are handled with Ratis and PipelineManger.
> The ReplicationManager should handle the replication of the ClosedContainers.
> The replication information will be sent as an event
> (UnderReplicated/OverReplicated).
> The Replication manager will collect all of the events in a priority queue
> (to replicate first the containers where more replica is missing) calculate
> the destination datanode (first with a very simple algorithm, later with
> calculating scatter-width) and send the Copy/Delete container to the datanode
> (CommandQueue).
> A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the
> copy/delete in case of failure. This is an in-memory structure (based on
> HDDS-195) which can requeue the underreplicated/overreplicated events to the
> prioirity queue unless the confirmation of the copy/delete command is arrived.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-199) Implement ReplicationManager to replicate ClosedContainers
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547420#comment-16547420 ] Elek, Marton commented on HDDS-199: --- Rebased patch is uploaded... > Implement ReplicationManager to replicate ClosedContainers > -- > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-199.001.patch, HDDS-199.002.patch, > HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch, > HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch, > HDDS-199.009.patch, HDDS-199.010.patch, HDDS-199.011.patch > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-199) Implement ReplicationManager to replicate ClosedContainers
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-199: -- Attachment: HDDS-199.011.patch > Implement ReplicationManager to replicate ClosedContainers > -- > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-199.001.patch, HDDS-199.002.patch, > HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch, > HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch, > HDDS-199.009.patch, HDDS-199.010.patch, HDDS-199.011.patch > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-199) Implement ReplicationManager to replicate ClosedContainers
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544532#comment-16544532 ] Elek, Marton commented on HDDS-199: --- Yes, but IMHO it's not a strict dependency. This patch can be committed as It won't break anything just the full replication won't work without HDDS-256. > Implement ReplicationManager to replicate ClosedContainers > -- > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-199.001.patch, HDDS-199.002.patch, > HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch, > HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch, > HDDS-199.009.patch, HDDS-199.010.patch > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-10) docker changes to test secure ozone cluster
[ https://issues.apache.org/jira/browse/HDDS-10?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554021#comment-16554021 ] Elek, Marton commented on HDDS-10: -- Thank you very much the update [~ajayydv] Still I don't understand what is the plan: 1. We can download the published binary from github (ADD https://github.com/flokkr/issuer/releases/download/1.0.0/issuer_1.0.0_linux_amd64.tar.gz /root/issuer) 2. Or we can include the source files and compile the helper application during the Docker image creation (with docker multiphase build) The last patch includes both the download of the binary and the sources (BTW without apache source header). I think we need only one of them. (I thinks just include the download is more simple but I amnot against to include the source and do the compilation during the docker build). > docker changes to test secure ozone cluster > --- > > Key: HDDS-10 > URL: https://issues.apache.org/jira/browse/HDDS-10 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Security >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.3.0 > > Attachments: HDDS-10-HDDS-4.00.patch, HDDS-10-HDDS-4.01.patch, > HDDS-10-HDDS-4.02.patch, HDDS-10-HDDS-4.03.patch > > > Update docker compose and settings to test secure ozone cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
Elek, Marton created HDDS-280:
-
Summary: Support ozone dist-start-stitching on openbsd/osx
Key: HDDS-280
URL: https://issues.apache.org/jira/browse/HDDS-280
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Elek, Marton
{quote}Ozone is creating a symlink during the dist process.
Using the "ozone" directory as a destination name all the docker-based
acceptance tests and docker-compose files are more simple as they don't need to
have the version information in the path.
But to keep the version specific folder name in the tar file we create a
symbolic link during the tar creation. With the symbolic link and the
'–dereference' tar argument we can create the tar file which includes a
versioned directory (ozone-0.2.1) but we can use the a dist directory without
the version in the name (hadoop-dist/target/ozone).
{quote}
This is the description of the current
dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
pointed to the problem that some bsd variants don't support the dereference
command line options of the ln command.
The main reason to use this approach is to get a simplified destination name
without the version (hadoop-dist/target/ozone instead of
hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
environments and acceptance tests, therefore I prefer to keep the simplified
destination name.
The issue is the tar file creation, if and only if we need the version number
in the name of the root directory inside of the tar.
Possible solutions:
# Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow and
requires more space.
# Do the tar distribution from docker all the time in case of 'dereference' is
not supported. Not very convenient
# Accept that tar will contain ozone directory and not ozone-0.2.1. This is
the more simple and can be improved with an additional VERSION file in the root
of the distribution.
# (+1) Use hadoop-dist/target/ozone-0.2.1 instead of hadoop-dist/target/ozone.
This is more complex for the docker based testing as we need the explicit names
in the compose files (volume: ../../../hadoop-dist/target/ozone-0.2.1). The
structure is more complex with using version in the directory name.
Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-280) Support ozone dist-start-stitching on openbsd/osx
[
https://issues.apache.org/jira/browse/HDDS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552463#comment-16552463
]
Elek, Marton commented on HDDS-280:
---
My preference is 3 with additional VERSION file.
> Support ozone dist-start-stitching on openbsd/osx
> -
>
> Key: HDDS-280
> URL: https://issues.apache.org/jira/browse/HDDS-280
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Priority: Major
>
> {quote}Ozone is creating a symlink during the dist process.
> Using the "ozone" directory as a destination name all the docker-based
> acceptance tests and docker-compose files are more simple as they don't need
> to have the version information in the path.
> But to keep the version specific folder name in the tar file we create a
> symbolic link during the tar creation. With the symbolic link and the
> '–dereference' tar argument we can create the tar file which includes a
> versioned directory (ozone-0.2.1) but we can use the a dist directory without
> the version in the name (hadoop-dist/target/ozone).
> {quote}
> This is the description of the current
> dev-support/bin/ozone-dist-tar-stitching. [~aw] in a comment for HDDS-276
> pointed to the problem that some bsd variants don't support the dereference
> command line options of the ln command.
> The main reason to use this approach is to get a simplified destination name
> without the version (hadoop-dist/target/ozone instead of
> hadoop-dist/target/ozone-0.2.1). It simplifies the docker-compose based
> environments and acceptance tests, therefore I prefer to keep the simplified
> destination name.
> The issue is the tar file creation, if and only if we need the version number
> in the name of the root directory inside of the tar.
> Possible solutions:
> # Use cp target/ozone target/ozone-0.2.1 + tar. It's simple but more slow
> and requires more space.
> # Do the tar distribution from docker all the time in case of 'dereference'
> is not supported. Not very convenient
> # Accept that tar will contain ozone directory and not ozone-0.2.1. This is
> the more simple and can be improved with an additional VERSION file in the
> root of the distribution.
> # (+1) Use hadoop-dist/target/ozone-0.2.1 instead of
> hadoop-dist/target/ozone. This is more complex for the docker based testing
> as we need the explicit names in the compose files (volume:
> ../../../hadoop-dist/target/ozone-0.2.1). The structure is more complex with
> using version in the directory name.
> Please comment your preference.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-276) Fix symbolic link creation during Ozone dist process
[ https://issues.apache.org/jira/browse/HDDS-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552465#comment-16552465 ] Elek, Marton commented on HDDS-276: --- Thank you very much the feedback [~aw] The mentioned behaviour is independent from this patch. This patch is adding just an additional check (an if condition) to work more stable. If I understood well you are -1 for the current implementation not for this patch. Thanks to point to this limitation (I checked the freebsd tar and it contains the _dereference_ option, but seems to be missing in openbsd and freebsd. I opened a separated issue for that (HDDS-280). I propose to commit this patch as it doesn't introduce and new linking (just fixes the existing one), unless the HDDS-280 will be committed in the next 3 days. [~aw] What do you think? > Fix symbolic link creation during Ozone dist process > > > Key: HDDS-276 > URL: https://issues.apache.org/jira/browse/HDDS-276 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.2.1 >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-276.001.patch > > > Ozone is creating a symlink during the dist process. > Using the "ozone" directory as a destination name all the docker-based > acceptance tests and docker-compose files are more simple as they don't need > to have the version information in the path. > But to keep the version specific folder name in the tar file we create a > symbolic link during the tar creation. With the symbolic link and the > '–dereference' tar argument we can create the tar file which includes a > versioned directory (ozone-0.2.1) but we can use the a dist directory without > the version in the name (hadoop-dist/target/ozone). > Currently this symlink creation has an issue. It couldn't be run twice. You > need to do a 'mvn clean' before you can create a new dist. > But fortunately this could be fixed easily by checking if the destination > symlink exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-199) Implement ReplicationManager to handle underreplication of closed containers
[ https://issues.apache.org/jira/browse/HDDS-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-199: -- Attachment: HDDS-199.013.patch > Implement ReplicationManager to handle underreplication of closed containers > > > Key: HDDS-199 > URL: https://issues.apache.org/jira/browse/HDDS-199 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-199.001.patch, HDDS-199.002.patch, > HDDS-199.003.patch, HDDS-199.004.patch, HDDS-199.005.patch, > HDDS-199.006.patch, HDDS-199.007.patch, HDDS-199.008.patch, > HDDS-199.009.patch, HDDS-199.010.patch, HDDS-199.011.patch, > HDDS-199.012.patch, HDDS-199.013.patch > > > HDDS/Ozone supports Open and Closed containers. In case of specific > conditions (container is full, node is failed) the container will be closed > and will be replicated in a different way. The replication of Open containers > are handled with Ratis and PipelineManger. > The ReplicationManager should handle the replication of the ClosedContainers. > The replication information will be sent as an event > (UnderReplicated/OverReplicated). > The Replication manager will collect all of the events in a priority queue > (to replicate first the containers where more replica is missing) calculate > the destination datanode (first with a very simple algorithm, later with > calculating scatter-width) and send the Copy/Delete container to the datanode > (CommandQueue). > A CopyCommandWatcher/DeleteCommandWatcher are also included to retry the > copy/delete in case of failure. This is an in-memory structure (based on > HDDS-195) which can requeue the underreplicated/overreplicated events to the > prioirity queue unless the confirmation of the copy/delete command is arrived. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
