[jira] [Updated] (HDDS-4404) Datanode can go OOM when a Recon or SCM Server is very slow in processing reports.
[ https://issues.apache.org/jira/browse/HDDS-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4404: Description: >From [~nanda619]'s analysis. ContainerReportPublisher thread runs periodically (default interval 60s) in Datanode and adds ContainerReport to StateContext (Queue). Heartbeat thread runs periodically (default interval 30s), picks up the ContainerReport (if any) from StateContext. For short time, the ContainerReport will be held in Datanode StateContext. For Recon, a change was made in datanode such that the ContainerReport will be cached in Datanode StateContext separately for each endpoint (i.e. SCM and Recon). As I see, if Recon is configured in the Datanode and all the reports that are to be sent to Recon will be pending in the StateContextQueue (LinkedList) > Datanode can go OOM when a Recon or SCM Server is very slow in processing > reports. > -- > > Key: HDDS-4404 > URL: https://issues.apache.org/jira/browse/HDDS-4404 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Datanode >Affects Versions: 1.0.0 >Reporter: Aravindan Vijayan >Priority: Critical > Attachments: Screen Shot 2020-10-26 at 11.24.09 PM.png > > > From [~nanda619]'s analysis. > ContainerReportPublisher thread runs periodically (default interval 60s) in > Datanode and adds ContainerReport to StateContext (Queue). > Heartbeat thread runs periodically (default interval 30s), picks up the > ContainerReport (if any) from StateContext. > For short time, the ContainerReport will be held in Datanode StateContext. > For Recon, a change was made in datanode such that the ContainerReport will > be cached in Datanode StateContext separately for each endpoint (i.e. SCM and > Recon). As I see, if Recon is configured in the Datanode and all the reports > that are to be sent to Recon will be pending in the StateContextQueue > (LinkedList) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4404) Datanode can go OOM when a Recon or SCM Server is very slow in processing reports.
Aravindan Vijayan created HDDS-4404: --- Summary: Datanode can go OOM when a Recon or SCM Server is very slow in processing reports. Key: HDDS-4404 URL: https://issues.apache.org/jira/browse/HDDS-4404 Project: Hadoop Distributed Data Store Issue Type: Task Components: Ozone Datanode Affects Versions: 1.0.0 Reporter: Aravindan Vijayan Attachments: Screen Shot 2020-10-26 at 11.24.09 PM.png -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4401) Fix compilation issue in HDDS-3698-upgrade branch.
[
https://issues.apache.org/jira/browse/HDDS-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4401:
Resolution: Fixed
Status: Resolved (was: Patch Available)
PR Merged.
> Fix compilation issue in HDDS-3698-upgrade branch.
> --
>
> Key: HDDS-4401
> URL: https://issues.apache.org/jira/browse/HDDS-4401
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-ozone-ozone-manager: Compilation failure: Compilation
> failure:
> Error:
> /mnt/ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:[691,6]
> not a statement
> Error:
> /mnt/ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:[691,23]
> ';' expected
> Error: -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4401) Fix compilation issue in HDDS-3698-upgrade branch.
[
https://issues.apache.org/jira/browse/HDDS-4401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4401:
Status: Patch Available (was: In Progress)
> Fix compilation issue in HDDS-3698-upgrade branch.
> --
>
> Key: HDDS-4401
> URL: https://issues.apache.org/jira/browse/HDDS-4401
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-ozone-ozone-manager: Compilation failure: Compilation
> failure:
> Error:
> /mnt/ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:[691,6]
> not a statement
> Error:
> /mnt/ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:[691,23]
> ';' expected
> Error: -> [Help 1]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4326) Investigate Acceptance test failure in Ozone Upgrade branch.
[
https://issues.apache.org/jira/browse/HDDS-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4326:
Description:
{code}
mapreduce-ofs :: Execute MR jobs
==
Execute PI calculation| PASS |
--
Execute WordCount | FAIL |
255 != 0
--
mapreduce-ofs :: Execute MR jobs | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
==
{code}
> Investigate Acceptance test failure in Ozone Upgrade branch.
>
>
> Key: HDDS-4326
> URL: https://issues.apache.org/jira/browse/HDDS-4326
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Affects Versions: 1.1.0
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Blocker
> Fix For: 1.1.0
>
>
> {code}
> mapreduce-ofs :: Execute MR jobs
> ==
> Execute PI calculation| PASS |
> --
> Execute WordCount | FAIL |
> 255 != 0
> --
> mapreduce-ofs :: Execute MR jobs | FAIL |
> 2 critical tests, 1 passed, 1 failed
> 2 tests total, 1 passed, 1 failed
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4403) Update the container replica history to the Recon DB lazily instead of for every report.
[ https://issues.apache.org/jira/browse/HDDS-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4403: Description: Recon tracks the history for every container replica on the Ozone cluster in its SQL DB (By default, this is Derby). To track this, it keeps track of the last timestamp of a replica on a DN through reports. This becomes a SQL DB scan + write operation for every container report received. Even though there is async hand off from the report to EventQueue, the event queue handler itself by default uses 1 thread per event type (report type). Hence, there is implicit blocking behavior here which is pushed down to DNs. This has to be changed into a lazy update of DB to support better scalability. Details on how to achieve this will be added to the JIRA later. was: Recon tracks the history for every container replica on the Ozone cluster. To track this, it keeps track of the last timestamp of a replica on a DN through reports. This becomes a SQL DB scan + write operation for every container report received. On a cluster with million containers, this amounts to 1 million per scan + write operations per minute. This has to be changed into a lazy update of DB to support better scalability. Details on how to achieve this will be added to the JIRA later. > Update the container replica history to the Recon DB lazily instead of for > every report. > > > Key: HDDS-4403 > URL: https://issues.apache.org/jira/browse/HDDS-4403 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > > Recon tracks the history for every container replica on the Ozone cluster in > its SQL DB (By default, this is Derby). To track this, it keeps track of the > last timestamp of a replica on a DN through reports. This becomes a SQL DB > scan + write operation for every container report received. Even though > there is async hand off from the report to EventQueue, the event queue > handler itself by default uses 1 thread per event type (report type). Hence, > there is implicit blocking behavior here which is pushed down to DNs. > This has to be changed into a lazy update of DB to support better > scalability. Details on how to achieve this will be added to the JIRA later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4403) Update the container replica history to the Recon DB lazily instead of for every report.
[ https://issues.apache.org/jira/browse/HDDS-4403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4403: Description: Recon tracks the history for every container replica on the Ozone cluster. To track this, it keeps track of the last timestamp of a replica on a DN through reports. This becomes a SQL DB scan + write operation for every container report received. On a cluster with million containers, this amounts to 1 million per scan + write operations per minute. This has to be changed into a lazy update of DB to support better scalability. Details on how to achieve this will be added to the JIRA later. was: Recon tracks the history for every container replica on the Ozone cluster. To track this, it keeps track of the last timestamp of a replica on a DN through reports. This becomes a SQL DB scan + write operation for every container report received (1 million per minute). This has to be changed into a lazy update of DB to support better scalability. Details on how to achieve this will be added to the JIRA later. > Update the container replica history to the Recon DB lazily instead of for > every report. > > > Key: HDDS-4403 > URL: https://issues.apache.org/jira/browse/HDDS-4403 > Project: Hadoop Distributed Data Store > Issue Type: Task >Reporter: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > > Recon tracks the history for every container replica on the Ozone cluster. To > track this, it keeps track of the last timestamp of a replica on a DN through > reports. This becomes a SQL DB scan + write operation for every container > report received. On a cluster with million containers, this amounts to 1 > million per scan + write operations per minute. > This has to be changed into a lazy update of DB to support better > scalability. Details on how to achieve this will be added to the JIRA later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4403) Update the container replica history to the Recon DB lazily instead of for every report.
Aravindan Vijayan created HDDS-4403: --- Summary: Update the container replica history to the Recon DB lazily instead of for every report. Key: HDDS-4403 URL: https://issues.apache.org/jira/browse/HDDS-4403 Project: Hadoop Distributed Data Store Issue Type: Task Reporter: Aravindan Vijayan Fix For: 1.1.0 Recon tracks the history for every container replica on the Ozone cluster. To track this, it keeps track of the last timestamp of a replica on a DN through reports. This becomes a SQL DB scan + write operation for every container report received (1 million per minute). This has to be changed into a lazy update of DB to support better scalability. Details on how to achieve this will be added to the JIRA later. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4402) Recon dashboard page does not load until missing containers API returns data.
Aravindan Vijayan created HDDS-4402: --- Summary: Recon dashboard page does not load until missing containers API returns data. Key: HDDS-4402 URL: https://issues.apache.org/jira/browse/HDDS-4402 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Recon Reporter: Aravindan Vijayan Assignee: Vivek Ratnavel Subramanian Fix For: 1.1.0 On a cluster with million containers, it was observed that the Recon dashboard page does not load even when most of the information has been returned by the backend through the /clusterState API, but the /missing containers API is still blocked. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4401) Fix compilation issue in HDDS-3698-upgrade branch.
Aravindan Vijayan created HDDS-4401:
---
Summary: Fix compilation issue in HDDS-3698-upgrade branch.
Key: HDDS-4401
URL: https://issues.apache.org/jira/browse/HDDS-4401
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Reporter: Aravindan Vijayan
Assignee: Aravindan Vijayan
Fix For: 1.1.0
{code}
Error: Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-ozone-ozone-manager: Compilation failure: Compilation failure:
Error:
/mnt/ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:[691,6]
not a statement
Error:
/mnt/ozone/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServer.java:[691,23]
';' expected
Error: -> [Help 1]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4386) Each EndpointStateMachine uses its own thread pool to talk with SCM/Recon
[
https://issues.apache.org/jira/browse/HDDS-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan resolved HDDS-4386.
-
Resolution: Fixed
Merged the PR. Thanks for the fix [~glengeng].
> Each EndpointStateMachine uses its own thread pool to talk with SCM/Recon
> -
>
> Key: HDDS-4386
> URL: https://issues.apache.org/jira/browse/HDDS-4386
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Glen Geng
>Assignee: Glen Geng
>Priority: Blocker
> Labels: pull-request-available
>
> In Tencent production environment, after start Recon for a while, we got
> warnings that all DNs become stale/dead at SCM side. After kill recon, all
> DNs become healthy in a very short time.
>
> *The root cause is:*
> 1) EndpointStateMachine for SCM and that for Recon share the thread pool
> created by DatanodeStateMachine, which is a fixed size thread pool:
> {code:java}
> executorService = Executors.newFixedThreadPool(
> getEndPointTaskThreadPoolSize(),
> new ThreadFactoryBuilder()
> .setNameFormat("Datanode State Machine Task Thread - %d").build());
> private int getEndPointTaskThreadPoolSize() {
> // TODO(runzhiwang): current only support one recon, if support multiple
> // recon in future reconServerCount should be the real number of recon
> int reconServerCount = 1;
> int totalServerCount = reconServerCount;
> try {
> totalServerCount += HddsUtils.getSCMAddresses(conf).size();
> } catch (Exception e) {
> LOG.error("Fail to get scm addresses", e);
> }
> return totalServerCount;
> }
> {code}
> meanwhile, current Recon has some performance issue, after running for hours,
> it became slower and slower, and crashed due to OOM.
> 2) The communication between DN and Recon will soon exhaust all the threads
> in DatanodeStateMachine.executorService, there will be no available threads
> for DN to talk SCM.
> 3) all DNs become stale/dead at SCM side.
>
> *The fix is quite straightforward:*
> Each EndpointStateMachine uses its own thread pool to talk with SCM/Recon, a
> slow Recon won't interfere the communication between DN and SCM, or vice
> versa.
>
> *P.S.*
> The first edition for DatanodeStateMachine.executorService is a cached thread
> pool, if there exists a slow SCM/Recon, more and more threads will be
> created, and DN will OOM eventually, due to tens of thousands of threads are
> created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4355) Deleted container is marked as missing on recon UI
[
https://issues.apache.org/jira/browse/HDDS-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221397#comment-17221397
]
Aravindan Vijayan commented on HDDS-4355:
-
[~Sammi] Yes, this is a known limitation in Recon currently.
> Deleted container is marked as missing on recon UI
> --
>
> Key: HDDS-4355
> URL: https://issues.apache.org/jira/browse/HDDS-4355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Priority: Major
> Attachments: screenshot-1.png
>
>
> {noformat}
> ~/ozoneenv/ozone]$ bin/ozone admin container info 104825
> Container id: 104825
> Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
> Container State: DELETED
> Datanodes: []
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4355) Deleted container is marked as missing on recon UI
[
https://issues.apache.org/jira/browse/HDDS-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan reassigned HDDS-4355:
---
Assignee: Aravindan Vijayan
> Deleted container is marked as missing on recon UI
> --
>
> Key: HDDS-4355
> URL: https://issues.apache.org/jira/browse/HDDS-4355
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Sammi Chen
>Assignee: Aravindan Vijayan
>Priority: Major
> Attachments: screenshot-1.png
>
>
> {noformat}
> ~/ozoneenv/ozone]$ bin/ozone admin container info 104825
> Container id: 104825
> Pipeline id: 10955a24-2047-416f-85ac-94523cfe8d40
> Container State: DELETED
> Datanodes: []
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4385) It would be nice if there is a search function using container ID on Recon Missing Container page
[ https://issues.apache.org/jira/browse/HDDS-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221390#comment-17221390 ] Aravindan Vijayan commented on HDDS-4385: - [~vivekratnavel] Assigning this to you since it is a UI request. > It would be nice if there is a search function using container ID on Recon > Missing Container page > - > > Key: HDDS-4385 > URL: https://issues.apache.org/jira/browse/HDDS-4385 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Recon >Reporter: Sammi Chen >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Attachments: image-2020-10-23-12-08-12-705.png > > > In production cluster, there can be many missing containers to investigate. > It would be nice to have a search filer using Container ID. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4385) It would be nice if there is a search function using container ID on Recon Missing Container page
[ https://issues.apache.org/jira/browse/HDDS-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4385: --- Assignee: Vivek Ratnavel Subramanian > It would be nice if there is a search function using container ID on Recon > Missing Container page > - > > Key: HDDS-4385 > URL: https://issues.apache.org/jira/browse/HDDS-4385 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Recon >Reporter: Sammi Chen >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Attachments: image-2020-10-23-12-08-12-705.png > > > In production cluster, there can be many missing containers to investigate. > It would be nice to have a search filer using Container ID. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4175) Implement Datanode Finalization
[ https://issues.apache.org/jira/browse/HDDS-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4175: --- Assignee: Prashant Pogde (was: István Fajth) > Implement Datanode Finalization > --- > > Key: HDDS-4175 > URL: https://issues.apache.org/jira/browse/HDDS-4175 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Fix For: 1.1.0 > > > * Create FinalizeCommand in SCM and Datanode protocol. > * Create FinalizeCommand Handler in Datanode. > * Datanode Finalization should FAIL if there are open containers on it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4175) Implement Datanode Finalization
[ https://issues.apache.org/jira/browse/HDDS-4175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221043#comment-17221043 ] Aravindan Vijayan commented on HDDS-4175: - [~ppogde] If this is already handled by other JIRAs, please close link related JIRAs and close as Duplicate. > Implement Datanode Finalization > --- > > Key: HDDS-4175 > URL: https://issues.apache.org/jira/browse/HDDS-4175 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Fix For: 1.1.0 > > > * Create FinalizeCommand in SCM and Datanode protocol. > * Create FinalizeCommand Handler in Datanode. > * Datanode Finalization should FAIL if there are open containers on it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4178) SCM Finalize command implementation.
[ https://issues.apache.org/jira/browse/HDDS-4178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4178: --- Assignee: Prashant Pogde (was: István Fajth) > SCM Finalize command implementation. > > > Key: HDDS-4178 > URL: https://issues.apache.org/jira/browse/HDDS-4178 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: SCM >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Fix For: 1.1.0 > > > * RPC endpoint implementation > * Ratis request to persist MLV, Trigger DN Finalize, Pipeline close. (WHEN > MLV changes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4182) Onboard HDDS-3869 into Layout version management
[ https://issues.apache.org/jira/browse/HDDS-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4182: --- Assignee: Ethan Rose > Onboard HDDS-3869 into Layout version management > > > Key: HDDS-4182 > URL: https://issues.apache.org/jira/browse/HDDS-4182 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Ethan Rose >Priority: Major > > In HDDS-3869 (Use different column families for datanode block and metadata), > there was a backward compatible change made in the Ozone datanode RocksDB. > This JIRA tracks the effort to use a "Layout Version" to track this change > such that it is NOT used before finalizing the cluster. > cc [~erose], [~hanishakoneru] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4314) OM Layout Version Manager init throws silent CNF error in integration tests.
[
https://issues.apache.org/jira/browse/HDDS-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4314:
Resolution: Fixed
Status: Resolved (was: Patch Available)
PR Merged.
> OM Layout Version Manager init throws silent CNF error in integration tests.
>
>
> Key: HDDS-4314
> URL: https://issues.apache.org/jira/browse/HDDS-4314
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> org.reflections.ReflectionsException: could not get type for name
> mockit.MockUp
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
> at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
> at org.reflections.Reflections.(Reflections.java:140)
> at org.reflections.Reflections.(Reflections.java:182)
> at org.reflections.Reflections.(Reflections.java:155)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.registerOzoneManagerRequests(OMLayoutVersionManagerImpl.java:122)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.init(OMLayoutVersionManagerImpl.java:100)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.initialize(OMLayoutVersionManagerImpl.java:83)
> at org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:363)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:930)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.createOMService(MiniOzoneHAClusterImpl.java:379)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.build(MiniOzoneHAClusterImpl.java:294)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.init(TestOzoneManagerHA.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.lang.ClassNotFoundException: mockit.MockUp
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
> ... 23 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4314) OM Layout Version Manager init throws silent CNF error in integration tests.
[
https://issues.apache.org/jira/browse/HDDS-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4314:
Assignee: Aravindan Vijayan
Status: Patch Available (was: Open)
> OM Layout Version Manager init throws silent CNF error in integration tests.
>
>
> Key: HDDS-4314
> URL: https://issues.apache.org/jira/browse/HDDS-4314
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Manager
>Reporter: Aravindan Vijayan
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> org.reflections.ReflectionsException: could not get type for name
> mockit.MockUp
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
> at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
> at org.reflections.Reflections.(Reflections.java:140)
> at org.reflections.Reflections.(Reflections.java:182)
> at org.reflections.Reflections.(Reflections.java:155)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.registerOzoneManagerRequests(OMLayoutVersionManagerImpl.java:122)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.init(OMLayoutVersionManagerImpl.java:100)
> at
> org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.initialize(OMLayoutVersionManagerImpl.java:83)
> at org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:363)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:930)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.createOMService(MiniOzoneHAClusterImpl.java:379)
> at
> org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.build(MiniOzoneHAClusterImpl.java:294)
> at
> org.apache.hadoop.ozone.om.TestOzoneManagerHA.init(TestOzoneManagerHA.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.lang.ClassNotFoundException: mockit.MockUp
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
> ... 23 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4172) Implement Finalize command in Ozone Manager server.
[ https://issues.apache.org/jira/browse/HDDS-4172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4172. - Resolution: Fixed PR Merged. > Implement Finalize command in Ozone Manager server. > --- > > Key: HDDS-4172 > URL: https://issues.apache.org/jira/browse/HDDS-4172 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Affects Versions: 1.1.0 >Reporter: Aravindan Vijayan >Assignee: István Fajth >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Using changes from HDDS-4141 and HDDS-3829, we can finish the OM finalization > logic by implementing the Ratis request to Finalize. > On the server side, this finalize command should update the internal Upgrade > state to "Finalized". This operation can be a No-Op if there are no layout > changes across an upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4326) Investigate Acceptance test failure in Ozone Upgrade branch.
Aravindan Vijayan created HDDS-4326: --- Summary: Investigate Acceptance test failure in Ozone Upgrade branch. Key: HDDS-4326 URL: https://issues.apache.org/jira/browse/HDDS-4326 Project: Hadoop Distributed Data Store Issue Type: Sub-task Affects Versions: 1.1.0 Reporter: Aravindan Vijayan Assignee: Aravindan Vijayan Fix For: 1.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4319) Compile error with Java 11
[
https://issues.apache.org/jira/browse/HDDS-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4319:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged Github PR.
> Compile error with Java 11
> --
>
> Key: HDDS-4319
> URL: https://issues.apache.org/jira/browse/HDDS-4319
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: build
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Major
> Labels: pull-request-available
>
> {code:title=https://github.com/apache/hadoop-ozone/runs/1217093596#step:6:5632}
> Error: Failed to execute goal
> org.codehaus.mojo:aspectj-maven-plugin:1.10:compile (default) on project
> hadoop-ozone-ozone-manager: Execution default of goal
> org.codehaus.mojo:aspectj-maven-plugin:1.10:compile failed: Plugin
> org.codehaus.mojo:aspectj-maven-plugin:1.10 or one of its dependencies could
> not be resolved: Could not find artifact com.sun:tools:jar:11.0.8 at
> specified path /opt/hostedtoolcache/jdk/11.0.8/x64/../lib/tools.jar -> [Help
> 1]
> {code}
> https://github.com/mojohaus/aspectj-maven-plugin/issues/24#issuecomment-419077658
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4314) OM Layout Version Manager init throws silent CNF error in integration tests.
Aravindan Vijayan created HDDS-4314:
---
Summary: OM Layout Version Manager init throws silent CNF error in
integration tests.
Key: HDDS-4314
URL: https://issues.apache.org/jira/browse/HDDS-4314
Project: Hadoop Distributed Data Store
Issue Type: Sub-task
Components: Ozone Manager
Reporter: Aravindan Vijayan
Fix For: 1.1.0
{code}
org.reflections.ReflectionsException: could not get type for name mockit.MockUp
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
at org.reflections.Reflections.(Reflections.java:140)
at org.reflections.Reflections.(Reflections.java:182)
at org.reflections.Reflections.(Reflections.java:155)
at
org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.registerOzoneManagerRequests(OMLayoutVersionManagerImpl.java:122)
at
org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.init(OMLayoutVersionManagerImpl.java:100)
at
org.apache.hadoop.ozone.om.upgrade.OMLayoutVersionManagerImpl.initialize(OMLayoutVersionManagerImpl.java:83)
at org.apache.hadoop.ozone.om.OzoneManager.(OzoneManager.java:363)
at
org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:930)
at
org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.createOMService(MiniOzoneHAClusterImpl.java:379)
at
org.apache.hadoop.ozone.MiniOzoneHAClusterImpl$Builder.build(MiniOzoneHAClusterImpl.java:294)
at
org.apache.hadoop.ozone.om.TestOzoneManagerHA.init(TestOzoneManagerHA.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.lang.ClassNotFoundException: mockit.MockUp
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
... 23 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4253) SCM changes to process Layout Info in register request/response
[ https://issues.apache.org/jira/browse/HDDS-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4253. - Resolution: Fixed PR Merged. > SCM changes to process Layout Info in register request/response > --- > > Key: HDDS-4253 > URL: https://issues.apache.org/jira/browse/HDDS-4253 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4253) SCM changes to process Layout Info in register request/response
[ https://issues.apache.org/jira/browse/HDDS-4253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4253: Description: Add LayoutVersion request/response for DN registration. > SCM changes to process Layout Info in register request/response > --- > > Key: HDDS-4253 > URL: https://issues.apache.org/jira/browse/HDDS-4253 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available > > Add LayoutVersion request/response for DN registration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4309) Fix inconsistent Recon config keys that start with "recon.om."
[
https://issues.apache.org/jira/browse/HDDS-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4309:
Priority: Minor (was: Major)
> Fix inconsistent Recon config keys that start with "recon.om."
> --
>
> Key: HDDS-4309
> URL: https://issues.apache.org/jira/browse/HDDS-4309
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Recon
>Affects Versions: 1.0.0
>Reporter: Aravindan Vijayan
>Priority: Minor
> Labels: newbie
>
> {code}
> hadoop-hdds/common/src/main/resources/ozone-default.xml
> 2318:recon.om.connection.request.timeout
> 2327:recon.om.connection.timeout
> 2336:recon.om.socket.timeout
> 2345:recon.om.snapshot.task.initial.delay
> 2353:recon.om.snapshot.task.interval.delay
> 2361:recon.om.snapshot.task.flush.param
> {code}
> These need to be deprecated and changed to "ozone.recon.om.<>".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4309) Fix inconsistent Recon config keys that start with "recon.om."
Aravindan Vijayan created HDDS-4309:
---
Summary: Fix inconsistent Recon config keys that start with
"recon.om."
Key: HDDS-4309
URL: https://issues.apache.org/jira/browse/HDDS-4309
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Recon
Affects Versions: 1.0.0
Reporter: Aravindan Vijayan
{code}
hadoop-hdds/common/src/main/resources/ozone-default.xml
2318:recon.om.connection.request.timeout
2327:recon.om.connection.timeout
2336:recon.om.socket.timeout
2345:recon.om.snapshot.task.initial.delay
2353:recon.om.snapshot.task.interval.delay
2361:recon.om.snapshot.task.flush.param
{code}
These need to be deprecated and changed to "ozone.recon.om.<>".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4309) Fix inconsistent Recon config keys that start with "recon.om."
[
https://issues.apache.org/jira/browse/HDDS-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-4309:
Labels: newbie (was: )
> Fix inconsistent Recon config keys that start with "recon.om."
> --
>
> Key: HDDS-4309
> URL: https://issues.apache.org/jira/browse/HDDS-4309
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Recon
>Affects Versions: 1.0.0
>Reporter: Aravindan Vijayan
>Priority: Major
> Labels: newbie
>
> {code}
> hadoop-hdds/common/src/main/resources/ozone-default.xml
> 2318:recon.om.connection.request.timeout
> 2327:recon.om.connection.timeout
> 2336:recon.om.socket.timeout
> 2345:recon.om.snapshot.task.initial.delay
> 2353:recon.om.snapshot.task.interval.delay
> 2361:recon.om.snapshot.task.flush.param
> {code}
> These need to be deprecated and changed to "ozone.recon.om.<>".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4280) Document notable configurations for Recon
[ https://issues.apache.org/jira/browse/HDDS-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4280: Resolution: Fixed Status: Resolved (was: Patch Available) Merged the PR. > Document notable configurations for Recon > -- > > Key: HDDS-4280 > URL: https://issues.apache.org/jira/browse/HDDS-4280 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Recon >Affects Versions: 1.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Minor > Labels: pull-request-available > > In [Reon doc > link|https://hadoop.apache.org/ozone/docs/1.0.0/feature/recon.html], there is > no helpful description about how to quickly setup the Recon server. As Recon > is one major feature in Ozone 1.0 version, we need to completed this document. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4227) Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine.
[ https://issues.apache.org/jira/browse/HDDS-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4227: Resolution: Fixed Status: Resolved (was: Patch Available) Merged PR. > Implement a "prepareForUpgrade" step that applies all committed transactions > onto the OM state machine. > --- > > Key: HDDS-4227 > URL: https://issues.apache.org/jira/browse/HDDS-4227 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > *Why is this needed?* > Through HDDS-4143, we have a generic factory to handle multiple versions of > apply transaction implementations based on layout version. Hence, this > factory can be used to handle versioned requests across layout versions, > whenever both the versions need to exist in the code (Let's say for > HDDS-2939). > However, it has been noticed that the OM ratis requests are still undergoing > lot of minor changes (HDDS-4007, HDDS-4007, HDDS-3903), and in these cases it > will become hard to maintain 2 versions of the code just to support clean > upgrades. > Hence, the plan is to build a pre-upgrade utility (client API) that makes > sure that an OM instance has no "un-applied" transactions in this Raft log. > Invoking this client API makes sure that the upgrade starts with a clean > state. Of course, this would be needed only in a HA setup. In a non HA setup, > this can either be skipped, or when invoked will be a No-Op (Non Ratis) or > cause no harm (Single node Ratis). > *How does it work?* > Before updating the software bits, our goal is to get OMs to get to the > latest state with respect to apply transaction. The reason we want this is to > make sure that the same version of the code executes the AT step in all the 3 > OMs. In a high level, the flow will be as follows. > * Before upgrade, *stop* the OMs. > * Start OMs with a special flag --prepareUpgrade (This is something like > --init, which is a special state which stops the ephemeral OM instance after > doing some work) > * When OM is started with the --prepareUpgrade flag, it does not start the > RPC server, so no new requests can get in. > * In this state, we give every OM time to apply txn until the last txn. > * We know that at least 2 OMs would have gotten the last client request > transaction committed into their log. Hence, those 2 OMs are expected to > apply transaction to that index faster. > * At every OM, the Raft log will be purged after this wait period (so that > the replay does not happen), and a Ratis snapshot taken at last txn. > * Even if there is a lagger OM which is unable to get to last applied txn > index, its logs will be purged after the wait time expires. > * Now when OMs are started with newer version, all the OMs will start using > the new code. > * The lagger OM will get the new Ratis snapshot since there are no logs to > replay from. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4252) Add the current layout versions to DN - SCM proto payload.
[ https://issues.apache.org/jira/browse/HDDS-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4252: Fix Version/s: 1.1.0 > Add the current layout versions to DN - SCM proto payload. > -- > > Key: HDDS-4252 > URL: https://issues.apache.org/jira/browse/HDDS-4252 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4252) Add the current layout versions to DN - SCM proto payload.
[ https://issues.apache.org/jira/browse/HDDS-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4252. - Resolution: Fixed PR Merged. [~ppogde] Can you add some description to this JIRA? > Add the current layout versions to DN - SCM proto payload. > -- > > Key: HDDS-4252 > URL: https://issues.apache.org/jira/browse/HDDS-4252 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Prashant Pogde >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4182) Onboard HDDS-3869 into Layout version management
[ https://issues.apache.org/jira/browse/HDDS-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4182: Description: In HDDS-3869 (Use different column families for datanode block and metadata), there was a backward compatible change made in the Ozone datanode RocksDB. This JIRA tracks the effort to use a "Layout Version" to track this change such that it is NOT used before finalizing the cluster. cc [~erose], [~hanishakoneru] was: In HDDS-3869 (Use different column families for datanode block and metadata), there was a backward compatible change made in the Ozone datanode RocksDB. This JIRA tracks the effort to use a "Layout Version" to track this change such that it is NOT used before finalizing the cluster. cc [~erose], [~hkoneru] > Onboard HDDS-3869 into Layout version management > > > Key: HDDS-4182 > URL: https://issues.apache.org/jira/browse/HDDS-4182 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Priority: Major > > In HDDS-3869 (Use different column families for datanode block and metadata), > there was a backward compatible change made in the Ozone datanode RocksDB. > This JIRA tracks the effort to use a "Layout Version" to track this change > such that it is NOT used before finalizing the cluster. > cc [~erose], [~hanishakoneru] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4227) Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine.
[ https://issues.apache.org/jira/browse/HDDS-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4227: Status: Patch Available (was: In Progress) > Implement a "prepareForUpgrade" step that applies all committed transactions > onto the OM state machine. > --- > > Key: HDDS-4227 > URL: https://issues.apache.org/jira/browse/HDDS-4227 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > *Why is this needed?* > Through HDDS-4143, we have a generic factory to handle multiple versions of > apply transaction implementations based on layout version. Hence, this > factory can be used to handle versioned requests across layout versions, > whenever both the versions need to exist in the code (Let's say for > HDDS-2939). > However, it has been noticed that the OM ratis requests are still undergoing > lot of minor changes (HDDS-4007, HDDS-4007, HDDS-3903), and in these cases it > will become hard to maintain 2 versions of the code just to support clean > upgrades. > Hence, the plan is to build a pre-upgrade utility (client API) that makes > sure that an OM instance has no "un-applied" transactions in this Raft log. > Invoking this client API makes sure that the upgrade starts with a clean > state. Of course, this would be needed only in a HA setup. In a non HA setup, > this can either be skipped, or when invoked will be a No-Op (Non Ratis) or > cause no harm (Single node Ratis). > *How does it work?* > Before updating the software bits, our goal is to get OMs to get to the > latest state with respect to apply transaction. The reason we want this is to > make sure that the same version of the code executes the AT step in all the 3 > OMs. In a high level, the flow will be as follows. > * Before upgrade, *stop* the OMs. > * Start OMs with a special flag --prepareUpgrade (This is something like > --init, which is a special state which stops the ephemeral OM instance after > doing some work) > * When OM is started with the --prepareUpgrade flag, it does not start the > RPC server, so no new requests can get in. > * In this state, we give every OM time to apply txn until the last txn. > * We know that at least 2 OMs would have gotten the last client request > transaction committed into their log. Hence, those 2 OMs are expected to > apply transaction to that index faster. > * At every OM, the Raft log will be purged after this wait period (so that > the replay does not happen), and a Ratis snapshot taken at last txn. > * Even if there is a lagger OM which is unable to get to last applied txn > index, its logs will be purged after the wait time expires. > * Now when OMs are started with newer version, all the OMs will start using > the new code. > * The lagger OM will get the new Ratis snapshot since there are no logs to > replay from. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3698) Ozone Non-Rolling upgrades
[ https://issues.apache.org/jira/browse/HDDS-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3698: Status: Patch Available (was: Open) > Ozone Non-Rolling upgrades > -- > > Key: HDDS-3698 > URL: https://issues.apache.org/jira/browse/HDDS-3698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Attachments: Ozone Non-Rolling Upgrades (Presentation).pdf, Ozone > Non-Rolling Upgrades Doc v1.1.pdf, Ozone Non-Rolling Upgrades.pdf > > > Support for Non-rolling upgrades in Ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3698) Ozone Non-Rolling upgrades
[ https://issues.apache.org/jira/browse/HDDS-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3698: Status: In Progress (was: Patch Available) > Ozone Non-Rolling upgrades > -- > > Key: HDDS-3698 > URL: https://issues.apache.org/jira/browse/HDDS-3698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Attachments: Ozone Non-Rolling Upgrades (Presentation).pdf, Ozone > Non-Rolling Upgrades Doc v1.1.pdf, Ozone Non-Rolling Upgrades.pdf > > > Support for Non-rolling upgrades in Ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4182) Onboard HDDS-3869 into Layout version management
[ https://issues.apache.org/jira/browse/HDDS-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4182: --- Assignee: (was: Aravindan Vijayan) > Onboard HDDS-3869 into Layout version management > > > Key: HDDS-4182 > URL: https://issues.apache.org/jira/browse/HDDS-4182 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Priority: Major > > In HDDS-3869 (Use different column families for datanode block and metadata), > there was a backward compatible change made in the Ozone datanode RocksDB. > This JIRA tracks the effort to use a "Layout Version" to track this change > such that it is NOT used before finalizing the cluster. > cc [~erose], [~hkoneru] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4268) Prepare for Upgrade step should purge the log after waiting for the last txn to be applied.
[ https://issues.apache.org/jira/browse/HDDS-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4268: --- Assignee: Aravindan Vijayan > Prepare for Upgrade step should purge the log after waiting for the last txn > to be applied. > --- > > Key: HDDS-4268 > URL: https://issues.apache.org/jira/browse/HDDS-4268 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > This is a follow up task from HDDS-4227 in which the prepare > upgrade/downgrade task should purge the Raft log immediately after waiting > for the last txn to be applied. This is to make sure that we dont "apply" > transactions in different versions of the code across the quorum. A lagging > follower will use a Ratis snapshot to bootstrap itself on restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4268) Prepare for Upgrade step should purge the log after waiting for the last txn to be applied.
Aravindan Vijayan created HDDS-4268: --- Summary: Prepare for Upgrade step should purge the log after waiting for the last txn to be applied. Key: HDDS-4268 URL: https://issues.apache.org/jira/browse/HDDS-4268 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Aravindan Vijayan This is a follow up task from HDDS-4227 in which the prepare upgrade/downgrade task should purge the Raft log immediately after waiting for the last txn to be applied. This is to make sure that we dont "apply" transactions in different versions of the code across the quorum. A lagging follower will use a Ratis snapshot to bootstrap itself on restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3881) Add current layout version to OM Ratis Request
[ https://issues.apache.org/jira/browse/HDDS-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3881: Resolution: Fixed Status: Resolved (was: Patch Available) PR merged. > Add current layout version to OM Ratis Request > -- > > Key: HDDS-3881 > URL: https://issues.apache.org/jira/browse/HDDS-3881 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > To make sure the correct version of the applyTxn step is executed against the > request, we should add the version to the OM Request and use that version in > the applyTxn step. Add the current layout version (MLV) to the OM Ratis > request. If there is no layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a factory for OM Requests that returns an instance based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Resolution: Fixed Status: Resolved (was: Patch Available) PR merged. > Implement a factory for OM Requests that returns an instance based on layout > version. > - > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > * Add the current layout version (MLV) to the OM Ratis request. If there is > no layout version present, we can default to '0'. > * Implement Generic factory which stores different instances of Type 'T' > sharded by a key & version. A single key can be associated with different > versions of 'T'. This is to support a typical use case during upgrade to have > multiple versions of a class / method / object and chose them based on > current layout version at runtime. Before finalizing, an older version is > typically needed, and after finalize, a newer version is needed. > * Using the generic factory, we scan all the different OM "write" requests > and associate them with versions. > * Layout feature code refactoring. Added more comments and tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4174) Add current HDDS layout version to Datanode heartbeat and registration.
[ https://issues.apache.org/jira/browse/HDDS-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4174. - Resolution: Fixed PR merged. > Add current HDDS layout version to Datanode heartbeat and registration. > --- > > Key: HDDS-4174 > URL: https://issues.apache.org/jira/browse/HDDS-4174 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Add the layout version as a field to proto. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4039) Reduce the number of fields in hdds.proto to improve performance
[ https://issues.apache.org/jira/browse/HDDS-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4039. - Resolution: Fixed PR merged through Github. > Reduce the number of fields in hdds.proto to improve performance > > > Key: HDDS-4039 > URL: https://issues.apache.org/jira/browse/HDDS-4039 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Datanode >Affects Versions: 1.0.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: HuangTao >Priority: Major > Labels: pull-request-available > > HDDS-3989 introduced revision and buildDate fields to hdds.proto file. These > fields are required only for Recon UI and don't have to be part of hdds.proto. > Also, version and setupTime are other two fields which can be removed and > added only to the SCM registration type as per [~elek] > ([https://github.com/apache/hadoop-ozone/pull/1226|https://github.com/apache/hadoop-ozone/pull/1226#issuecomment-663416483]) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4039) Reduce the number of fields in hdds.proto to improve performance
[ https://issues.apache.org/jira/browse/HDDS-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4039: Fix Version/s: 1.1.0 > Reduce the number of fields in hdds.proto to improve performance > > > Key: HDDS-4039 > URL: https://issues.apache.org/jira/browse/HDDS-4039 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Datanode >Affects Versions: 1.0.0 >Reporter: Vivek Ratnavel Subramanian >Assignee: HuangTao >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > HDDS-3989 introduced revision and buildDate fields to hdds.proto file. These > fields are required only for Recon UI and don't have to be part of hdds.proto. > Also, version and setupTime are other two fields which can be removed and > added only to the SCM registration type as per [~elek] > ([https://github.com/apache/hadoop-ozone/pull/1226|https://github.com/apache/hadoop-ozone/pull/1226#issuecomment-663416483]) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4235) Ozone client FS path validation is not present in OFS.
Aravindan Vijayan created HDDS-4235: --- Summary: Ozone client FS path validation is not present in OFS. Key: HDDS-4235 URL: https://issues.apache.org/jira/browse/HDDS-4235 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Client Reporter: Aravindan Vijayan Fix For: 1.1.0 HDDS-3969 added a validation step in the BasicOzoneFileSystem which makes sure there are no keys with trailing ".." created through the o3fs API. This needs to go into the OFS code path as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4227) Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine.
[ https://issues.apache.org/jira/browse/HDDS-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4227: Description: *Why is this needed?* Through HDDS-4143, we have a generic factory to handle multiple versions of apply transaction implementations based on layout version. Hence, this factory can be used to handle versioned requests across layout versions, whenever both the versions need to exist in the code (Let's say for HDDS-2939). However, it has been noticed that the OM ratis requests are still undergoing lot of minor changes (HDDS-4007, HDDS-4007, HDDS-3903), and in these cases it will become hard to maintain 2 versions of the code just to support clean upgrades. Hence, the plan is to build a pre-upgrade utility (client API) that makes sure that an OM instance has no "un-applied" transactions in this Raft log. Invoking this client API makes sure that the upgrade starts with a clean state. Of course, this would be needed only in a HA setup. In a non HA setup, this can either be skipped, or when invoked will be a No-Op (Non Ratis) or cause no harm (Single node Ratis). *How does it work?* Before updating the software bits, our goal is to get OMs to get to the latest state with respect to apply transaction. The reason we want this is to make sure that the same version of the code executes the AT step in all the 3 OMs. In a high level, the flow will be as follows. * Before upgrade, *stop* the OMs. * Start OMs with a special flag --prepareUpgrade (This is something like --init, which is a special state which stops the ephemeral OM instance after doing some work) * When OM is started with the --prepareUpgrade flag, it does not start the RPC server, so no new requests can get in. * In this state, we give every OM time to apply txn until the last txn. * We know that at least 2 OMs would have gotten the last client request transaction committed into their log. Hence, those 2 OMs are expected to apply transaction to that index faster. * At every OM, the Raft log will be purged after this wait period (so that the replay does not happen), and a Ratis snapshot taken at last txn. * Even if there is a lagger OM which is unable to get to last applied txn index, its logs will be purged after the wait time expires. * Now when OMs are started with newer version, all the OMs will start using the new code. * The lagger OM will get the new Ratis snapshot since there are no logs to replay from. > Implement a "prepareForUpgrade" step that applies all committed transactions > onto the OM state machine. > --- > > Key: HDDS-4227 > URL: https://issues.apache.org/jira/browse/HDDS-4227 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > > *Why is this needed?* > Through HDDS-4143, we have a generic factory to handle multiple versions of > apply transaction implementations based on layout version. Hence, this > factory can be used to handle versioned requests across layout versions, > whenever both the versions need to exist in the code (Let's say for > HDDS-2939). > However, it has been noticed that the OM ratis requests are still undergoing > lot of minor changes (HDDS-4007, HDDS-4007, HDDS-3903), and in these cases it > will become hard to maintain 2 versions of the code just to support clean > upgrades. > Hence, the plan is to build a pre-upgrade utility (client API) that makes > sure that an OM instance has no "un-applied" transactions in this Raft log. > Invoking this client API makes sure that the upgrade starts with a clean > state. Of course, this would be needed only in a HA setup. In a non HA setup, > this can either be skipped, or when invoked will be a No-Op (Non Ratis) or > cause no harm (Single node Ratis). > *How does it work?* > Before updating the software bits, our goal is to get OMs to get to the > latest state with respect to apply transaction. The reason we want this is to > make sure that the same version of the code executes the AT step in all the 3 > OMs. In a high level, the flow will be as follows. > * Before upgrade, *stop* the OMs. > * Start OMs with a special flag --prepareUpgrade (This is something like > --init, which is a special state which stops the ephemeral OM instance after > doing some work) > * When OM is started with the --prepareUpgrade flag, it does not start the > RPC server, so no new requests can get in. > * In this state, we give every OM time to apply txn until the last txn. > * We know that at least 2 OMs would have gotten the last client request > transaction committed into their log.
[jira] [Created] (HDDS-4227) Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine.
Aravindan Vijayan created HDDS-4227: --- Summary: Implement a "prepareForUpgrade" step that applies all committed transactions onto the OM state machine. Key: HDDS-4227 URL: https://issues.apache.org/jira/browse/HDDS-4227 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Aravindan Vijayan Assignee: Aravindan Vijayan Fix For: 1.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a factory for OM Requests that returns an instance based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Description: * Add the current layout version (MLV) to the OM Ratis request. If there is no layout version present, we can default to '0'. * Implement Generic factory which stores different instances of Type 'T' sharded by a key & version. A single key can be associated with different versions of 'T'. This is to support a typical use case during upgrade to have multiple versions of a class / method / object and chose them based on current layout version at runtime. Before finalizing, an older version is typically needed, and after finalize, a newer version is needed. * Using the generic factory, we scan all the different OM "write" requests and associate them with versions. * Layout feature code refactoring. Added more comments and tests. was:Add the current layout version (MLV) to the OM Ratis request. If there is no layout version present, we can default to '0'. > Implement a factory for OM Requests that returns an instance based on layout > version. > - > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > * Add the current layout version (MLV) to the OM Ratis request. If there is > no layout version present, we can default to '0'. > * Implement Generic factory which stores different instances of Type 'T' > sharded by a key & version. A single key can be associated with different > versions of 'T'. This is to support a typical use case during upgrade to have > multiple versions of a class / method / object and chose them based on > current layout version at runtime. Before finalizing, an older version is > typically needed, and after finalize, a newer version is needed. > * Using the generic factory, we scan all the different OM "write" requests > and associate them with versions. > * Layout feature code refactoring. Added more comments and tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4141) Implement Finalize command in Ozone Manager client.
[ https://issues.apache.org/jira/browse/HDDS-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4141. - Resolution: Fixed PR merged. > Implement Finalize command in Ozone Manager client. > --- > > Key: HDDS-4141 > URL: https://issues.apache.org/jira/browse/HDDS-4141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: István Fajth >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > * On the client side, add a new command to finalize OM through CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-4173) Implement HDDS Version management using the LayoutVersionManager interface.
[ https://issues.apache.org/jira/browse/HDDS-4173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan resolved HDDS-4173. - Resolution: Fixed PR merged through Github. > Implement HDDS Version management using the LayoutVersionManager interface. > --- > > Key: HDDS-4173 > URL: https://issues.apache.org/jira/browse/HDDS-4173 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Datanode, SCM >Affects Versions: 1.1.0 >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > * Create HDDS Layout Feature Catalog similar to the OM Layout Feature Catalog. > * Any layout change to SCM and Datanode needs to be recorded here as a Layout > Feature. > * This includes new SCM HA requests, new container layouts in DN etc. > * Create a HDDSLayoutVersionManager similar to OMLayoutVersionManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a factory for OM Requests that returns an instance based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Status: Patch Available (was: In Progress) > Implement a factory for OM Requests that returns an instance based on layout > version. > - > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Add the current layout version (MLV) to the OM Ratis request. If there is no > layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3881) Add current layout version to OM Ratis Request
[ https://issues.apache.org/jira/browse/HDDS-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3881: Status: Patch Available (was: In Progress) > Add current layout version to OM Ratis Request > -- > > Key: HDDS-3881 > URL: https://issues.apache.org/jira/browse/HDDS-3881 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > To make sure the correct version of the applyTxn step is executed against the > request, we should add the version to the OM Request and use that version in > the applyTxn step. Add the current layout version (MLV) to the OM Ratis > request. If there is no layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4219) Revisit 'static' nature of OM Layout Version Manager.
[ https://issues.apache.org/jira/browse/HDDS-4219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4219: Parent: HDDS-3698 Issue Type: Sub-task (was: Bug) > Revisit 'static' nature of OM Layout Version Manager. > - > > Key: HDDS-4219 > URL: https://issues.apache.org/jira/browse/HDDS-4219 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > > Investigate whether we can programmatically instantiate the OM Aspect so that > we can move away from static nature of OM Layout Version Manager. Moving away > from static behavior will help out with easy unit testing. > cc [~pifta] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4219) Revisit 'static' nature of OM Layout Version Manager.
Aravindan Vijayan created HDDS-4219: --- Summary: Revisit 'static' nature of OM Layout Version Manager. Key: HDDS-4219 URL: https://issues.apache.org/jira/browse/HDDS-4219 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Reporter: Aravindan Vijayan Fix For: 1.1.0 Investigate whether we can programmatically instantiate the OM Aspect so that we can move away from static nature of OM Layout Version Manager. Moving away from static behavior will help out with easy unit testing. cc [~pifta] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a version factory for OM Requests that returns an instance based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Summary: Implement a version factory for OM Requests that returns an instance based on layout version. (was: Implement a version factory for OM Apply Transaction that uses the implementation based on layout version.) > Implement a version factory for OM Requests that returns an instance based on > layout version. > - > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Add the current layout version (MLV) to the OM Ratis request. If there is no > layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a factory for OM Requests that returns an instance based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Summary: Implement a factory for OM Requests that returns an instance based on layout version. (was: Implement a version factory for OM Requests that returns an instance based on layout version.) > Implement a factory for OM Requests that returns an instance based on layout > version. > - > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > Add the current layout version (MLV) to the OM Ratis request. If there is no > layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-3804) Recon start fails with SQL exception with MySQL DB.
[
https://issues.apache.org/jira/browse/HDDS-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan resolved HDDS-3804.
-
Resolution: Fixed
Thanks for fixing this [~flirmnave]. PR merged.
> Recon start fails with SQL exception with MySQL DB.
> ---
>
> Key: HDDS-3804
> URL: https://issues.apache.org/jira/browse/HDDS-3804
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Recon
>Affects Versions: 1.0.0
>Reporter: Aravindan Vijayan
>Assignee: Zheng Huang-Mu
>Priority: Major
> Labels: pull-request-available
>
> {code}
> org.jooq.exception.DataAccessException: SQL [create table if not exists
> `CONTAINER_HISTORY`(`container_id` bigint null, `datanode_host` varchar(1024)
> null, `first_report_timestamp` bigint null, `last_report_timestamp` bigint
> null, constraint `pk_container_id_datanode_host` primary key (`container_id`,
> `datanode_host`))]; Specified key was too long; max key length is 3072 bytes
> at org.jooq_3.11.9.MYSQL_5_7.debug(Unknown Source)
> at org.jooq.impl.Tools.translate(Tools.java:2429)
> at
> org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:832)
> at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:364)
> at
> org.hadoop.ozone.recon.schema.ContainerSchemaDefinition.createContainerHistoryTable(ContainerSchemaDefinition.java:90)
> at
> org.hadoop.ozone.recon.schema.ContainerSchemaDefinition.initializeSchema(ContainerSchemaDefinition.java:72)
> at
> org.apache.hadoop.ozone.recon.ReconSchemaManager.lambda$createReconSchema$0(ReconSchemaManager.java:50)
> at java.lang.Iterable.forEach(Iterable.java:75)
> at
> org.apache.hadoop.ozone.recon.ReconSchemaManager.createReconSchema(ReconSchemaManager.java:48)
> at
> org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:102)
> at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:53)
> at picocli.CommandLine.execute(CommandLine.java:1173)
> at picocli.CommandLine.access$800(CommandLine.java:141)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
> at
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:75)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:66)
> at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:67)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
> Specified key was too long; max key length is 3072 bytes
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
> at com.mysql.jdbc.Util.getInstance(Util.java:408)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:943)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2487)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
> at
> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1197)
> at
> com.jolbox.bonecp.PreparedStatementHandle.execute(PreparedStatementHandle.java:140)
> at
> org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:209)
> at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:432)
> at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:350)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3804) Recon start fails with SQL exception with MySQL DB.
[
https://issues.apache.org/jira/browse/HDDS-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aravindan Vijayan updated HDDS-3804:
Fix Version/s: 1.1.0
> Recon start fails with SQL exception with MySQL DB.
> ---
>
> Key: HDDS-3804
> URL: https://issues.apache.org/jira/browse/HDDS-3804
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Recon
>Affects Versions: 1.0.0
>Reporter: Aravindan Vijayan
>Assignee: Zheng Huang-Mu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.1.0
>
>
> {code}
> org.jooq.exception.DataAccessException: SQL [create table if not exists
> `CONTAINER_HISTORY`(`container_id` bigint null, `datanode_host` varchar(1024)
> null, `first_report_timestamp` bigint null, `last_report_timestamp` bigint
> null, constraint `pk_container_id_datanode_host` primary key (`container_id`,
> `datanode_host`))]; Specified key was too long; max key length is 3072 bytes
> at org.jooq_3.11.9.MYSQL_5_7.debug(Unknown Source)
> at org.jooq.impl.Tools.translate(Tools.java:2429)
> at
> org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:832)
> at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:364)
> at
> org.hadoop.ozone.recon.schema.ContainerSchemaDefinition.createContainerHistoryTable(ContainerSchemaDefinition.java:90)
> at
> org.hadoop.ozone.recon.schema.ContainerSchemaDefinition.initializeSchema(ContainerSchemaDefinition.java:72)
> at
> org.apache.hadoop.ozone.recon.ReconSchemaManager.lambda$createReconSchema$0(ReconSchemaManager.java:50)
> at java.lang.Iterable.forEach(Iterable.java:75)
> at
> org.apache.hadoop.ozone.recon.ReconSchemaManager.createReconSchema(ReconSchemaManager.java:48)
> at
> org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:102)
> at org.apache.hadoop.ozone.recon.ReconServer.call(ReconServer.java:53)
> at picocli.CommandLine.execute(CommandLine.java:1173)
> at picocli.CommandLine.access$800(CommandLine.java:141)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1367)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:1335)
> at
> picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1243)
> at picocli.CommandLine.parseWithHandlers(CommandLine.java:1526)
> at picocli.CommandLine.parseWithHandler(CommandLine.java:1465)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:75)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:66)
> at org.apache.hadoop.ozone.recon.ReconServer.main(ReconServer.java:67)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
> Specified key was too long; max key length is 3072 bytes
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
> at com.mysql.jdbc.Util.getInstance(Util.java:408)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:943)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2487)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
> at
> com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1197)
> at
> com.jolbox.bonecp.PreparedStatementHandle.execute(PreparedStatementHandle.java:140)
> at
> org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:209)
> at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:432)
> at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:350)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-4181) Add acceptance tests for upgrade, finalization and downgrade
[ https://issues.apache.org/jira/browse/HDDS-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188636#comment-17188636 ] Aravindan Vijayan commented on HDDS-4181: - Thanks [~elek]. I have seen that changes, and planning to build on top of that. > Add acceptance tests for upgrade, finalization and downgrade > > > Key: HDDS-4181 > URL: https://issues.apache.org/jira/browse/HDDS-4181 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-4181) Add acceptance tests for upgrade, finalization and downgrade
[ https://issues.apache.org/jira/browse/HDDS-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188636#comment-17188636 ] Aravindan Vijayan edited comment on HDDS-4181 at 9/1/20, 4:49 PM: -- Thanks [~elek]. I have seen those changes, and planning to build on top of that. was (Author: avijayan): Thanks [~elek]. I have seen that changes, and planning to build on top of that. > Add acceptance tests for upgrade, finalization and downgrade > > > Key: HDDS-4181 > URL: https://issues.apache.org/jira/browse/HDDS-4181 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3698) Ozone Non-Rolling upgrades
[ https://issues.apache.org/jira/browse/HDDS-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3698: Summary: Ozone Non-Rolling upgrades (was: Ozone Non-Rolling upgrades.) > Ozone Non-Rolling upgrades > -- > > Key: HDDS-3698 > URL: https://issues.apache.org/jira/browse/HDDS-3698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Attachments: Ozone Non-Rolling Upgrades (Presentation).pdf, Ozone > Non-Rolling Upgrades Doc v1.1.pdf, Ozone Non-Rolling Upgrades.pdf > > > Support for Non-rolling upgrades in Ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4183) Support backward compatible upgrade to a version with key prefix management.
[ https://issues.apache.org/jira/browse/HDDS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4183: Description: HDDS-2939 makes fundamental changes to how OM writes & reads its metadata. Since this is about as big a layout change as can be, we have to make sure this is properly managed through the upgrade flow. In addition to upgrades, we should support downgrades from pre-finalized state of OM. cc [~rakeshr] / [~msingh] > Support backward compatible upgrade to a version with key prefix management. > > > Key: HDDS-4183 > URL: https://issues.apache.org/jira/browse/HDDS-4183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > > HDDS-2939 makes fundamental changes to how OM writes & reads its metadata. > Since this is about as big a layout change as can be, we have to make sure > this is properly managed through the upgrade flow. In addition to upgrades, > we should support downgrades from pre-finalized state of OM. > cc [~rakeshr] / [~msingh] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4183) Support backward compatible upgrade to a version with key prefix management
[ https://issues.apache.org/jira/browse/HDDS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4183: --- Assignee: (was: Aravindan Vijayan) > Support backward compatible upgrade to a version with key prefix management > --- > > Key: HDDS-4183 > URL: https://issues.apache.org/jira/browse/HDDS-4183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > > HDDS-2939 makes fundamental changes to how OM writes & reads its metadata. > Since this is about as big a layout change as can be, we have to make sure > this is properly managed through the upgrade flow. In addition to upgrades, > we should support downgrades from pre-finalized state of OM. > cc [~rakeshr] / [~msingh] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4183) Support backward compatible upgrade to a version with key prefix management
[ https://issues.apache.org/jira/browse/HDDS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4183: Summary: Support backward compatible upgrade to a version with key prefix management (was: Support backward compatible upgrade to a version with key prefix management.) > Support backward compatible upgrade to a version with key prefix management > --- > > Key: HDDS-4183 > URL: https://issues.apache.org/jira/browse/HDDS-4183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > > HDDS-2939 makes fundamental changes to how OM writes & reads its metadata. > Since this is about as big a layout change as can be, we have to make sure > this is properly managed through the upgrade flow. In addition to upgrades, > we should support downgrades from pre-finalized state of OM. > cc [~rakeshr] / [~msingh] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4183) Support backward compatible upgrade to a version with key prefix management.
[ https://issues.apache.org/jira/browse/HDDS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4183: --- Assignee: Aravindan Vijayan > Support backward compatible upgrade to a version with key prefix management. > > > Key: HDDS-4183 > URL: https://issues.apache.org/jira/browse/HDDS-4183 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Fix For: 1.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4183) Support backward compatible upgrade to a version with key prefix management.
Aravindan Vijayan created HDDS-4183: --- Summary: Support backward compatible upgrade to a version with key prefix management. Key: HDDS-4183 URL: https://issues.apache.org/jira/browse/HDDS-4183 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Aravindan Vijayan Fix For: 1.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4182) Onboard HDDS-3869 into Layout version management
[ https://issues.apache.org/jira/browse/HDDS-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4182: Parent: HDDS-3698 Issue Type: Sub-task (was: Bug) > Onboard HDDS-3869 into Layout version management > > > Key: HDDS-4182 > URL: https://issues.apache.org/jira/browse/HDDS-4182 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > In HDDS-3869 (Use different column families for datanode block and metadata), > there was a backward compatible change made in the Ozone datanode RocksDB. > This JIRA tracks the effort to use a "Layout Version" to track this change > such that it is NOT used before finalizing the cluster. > cc [~erose], [~hkoneru] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4182) Onboard HDDS-3869 into Layout version management
Aravindan Vijayan created HDDS-4182: --- Summary: Onboard HDDS-3869 into Layout version management Key: HDDS-4182 URL: https://issues.apache.org/jira/browse/HDDS-4182 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Aravindan Vijayan In HDDS-3869 (Use different column families for datanode block and metadata), there was a backward compatible change made in the Ozone datanode RocksDB. This JIRA tracks the effort to use a "Layout Version" to track this change such that it is NOT used before finalizing the cluster. cc [~erose], [~hkoneru] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4182) Onboard HDDS-3869 into Layout version management
[ https://issues.apache.org/jira/browse/HDDS-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4182: --- Assignee: Aravindan Vijayan > Onboard HDDS-3869 into Layout version management > > > Key: HDDS-4182 > URL: https://issues.apache.org/jira/browse/HDDS-4182 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > In HDDS-3869 (Use different column families for datanode block and metadata), > there was a backward compatible change made in the Ozone datanode RocksDB. > This JIRA tracks the effort to use a "Layout Version" to track this change > such that it is NOT used before finalizing the cluster. > cc [~erose], [~hkoneru] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4181) Add acceptance tests for upgrade, finalization and downgrade
Aravindan Vijayan created HDDS-4181: --- Summary: Add acceptance tests for upgrade, finalization and downgrade Key: HDDS-4181 URL: https://issues.apache.org/jira/browse/HDDS-4181 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Aravindan Vijayan Fix For: 1.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4180) Container Layout Version Management in Datanode
Aravindan Vijayan created HDDS-4180: --- Summary: Container Layout Version Management in Datanode Key: HDDS-4180 URL: https://issues.apache.org/jira/browse/HDDS-4180 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Aravindan Vijayan * Chunk Layout Version persistence in Datanode * Add versioned read of Container Metadata File in Datanode -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4179) Implement post-finalize SCM logic to allow nodes of only new version to participate in pipelines.
Aravindan Vijayan created HDDS-4179: --- Summary: Implement post-finalize SCM logic to allow nodes of only new version to participate in pipelines. Key: HDDS-4179 URL: https://issues.apache.org/jira/browse/HDDS-4179 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Datanode, SCM Reporter: Aravindan Vijayan Fix For: 1.1.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4178) SCM Finalize command implementation.
Aravindan Vijayan created HDDS-4178: --- Summary: SCM Finalize command implementation. Key: HDDS-4178 URL: https://issues.apache.org/jira/browse/HDDS-4178 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Reporter: Aravindan Vijayan Fix For: 1.1.0 * RPC endpoint implementation * Ratis request to persist MLV, Trigger DN Finalize, Pipeline close. (WHEN MLV changes) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4175) Implement Datanode Finalization
Aravindan Vijayan created HDDS-4175: --- Summary: Implement Datanode Finalization Key: HDDS-4175 URL: https://issues.apache.org/jira/browse/HDDS-4175 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Datanode Reporter: Aravindan Vijayan Fix For: 1.1.0 * Create FinalizeCommand in SCM and Datanode protocol. * Create FinalizeCommand Handler in Datanode. * Datanode Finalization should FAIL if there are open containers on it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4174) Add current HDDS layout version to Datanode heartbeat and registration.
Aravindan Vijayan created HDDS-4174: --- Summary: Add current HDDS layout version to Datanode heartbeat and registration. Key: HDDS-4174 URL: https://issues.apache.org/jira/browse/HDDS-4174 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Datanode Reporter: Aravindan Vijayan Fix For: 1.1.0 Add the layout version as a field to proto. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4173) Implement HDDS Version management using the LayoutVersionManager interface.
Aravindan Vijayan created HDDS-4173: --- Summary: Implement HDDS Version management using the LayoutVersionManager interface. Key: HDDS-4173 URL: https://issues.apache.org/jira/browse/HDDS-4173 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Datanode, SCM Affects Versions: 1.1.0 Reporter: Aravindan Vijayan Fix For: 1.1.0 * Create HDDS Layout Feature Catalog similar to the OM Layout Feature Catalog. * Any layout change to SCM and Datanode needs to be recorded here as a Layout Feature. * This includes new SCM HA requests, new container layouts in DN etc. * Create a HDDSLayoutVersionManager similar to OMLayoutVersionManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4172) Implement Finalize command in Ozone Manager server.
Aravindan Vijayan created HDDS-4172: --- Summary: Implement Finalize command in Ozone Manager server. Key: HDDS-4172 URL: https://issues.apache.org/jira/browse/HDDS-4172 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Affects Versions: 0.7.0 Reporter: Aravindan Vijayan Assignee: Istvan Fajth Fix For: 0.7.0 Using changes from HDDS-4141 and HDDS-3829, we can finish the OM finalization logic by implementing the Ratis request to Finalize. On the server side, this finalize command should update the internal Upgrade state to "Finalized". This operation can be a No-Op if there are no layout changes across an upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4141) Implement Finalize command in Ozone Manager client.
[ https://issues.apache.org/jira/browse/HDDS-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4141: Description: * On the client side, add a new command to finalize OM through CLI. (was: * On the client side, add a new command to finalize OM through CLI. * On the server side, this finalize command should update the internal Upgrade state to "Finalized". This operation can be a No-Op if there are no layout changes across an upgrade.) > Implement Finalize command in Ozone Manager client. > --- > > Key: HDDS-4141 > URL: https://issues.apache.org/jira/browse/HDDS-4141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Istvan Fajth >Priority: Major > Fix For: 0.7.0 > > > * On the client side, add a new command to finalize OM through CLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4141) Implement Finalize command in Ozone Manager client.
[ https://issues.apache.org/jira/browse/HDDS-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4141: Summary: Implement Finalize command in Ozone Manager client. (was: Implement Finalize command in Ozone Manager) > Implement Finalize command in Ozone Manager client. > --- > > Key: HDDS-4141 > URL: https://issues.apache.org/jira/browse/HDDS-4141 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Istvan Fajth >Priority: Major > Fix For: 0.7.0 > > > * On the client side, add a new command to finalize OM through CLI. > * On the server side, this finalize command should update the internal > Upgrade state to "Finalized". This operation can be a No-Op if there are no > layout changes across an upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3829) Introduce Layout Feature interface in Ozone
[ https://issues.apache.org/jira/browse/HDDS-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3829: Resolution: Fixed Status: Resolved (was: Patch Available) > Introduce Layout Feature interface in Ozone > --- > > Key: HDDS-3829 > URL: https://issues.apache.org/jira/browse/HDDS-3829 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3829) Introduce Layout Feature interface in Ozone
[ https://issues.apache.org/jira/browse/HDDS-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3829: Description: * Implement the concept of a 'Layout Feature' in Ozone (with sample usage in Ozone Manager), which defines a specific change in on-disk layout in Ozone. * Every feature is associated with a layout version, and an API corresponding to the feature cannot be invoked (throws NOT_SUPPORTED_OPERATION) before finalization. * Created an annotation based 'aspect' for "guarding" new APIs that are introduced by Layout Features. Check out TestOMLayoutFeatureAspect#testCheckLayoutFeature. * Added sample features and tests for ease of review (To be removed before commit). * Created an abstract VersionManager and an inherited OM Version manager to initialize features, check if feature is allowed, check need to finalize, do finalization. > Introduce Layout Feature interface in Ozone > --- > > Key: HDDS-3829 > URL: https://issues.apache.org/jira/browse/HDDS-3829 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > > * Implement the concept of a 'Layout Feature' in Ozone (with sample usage in > Ozone Manager), which defines a specific change in on-disk layout in Ozone. > * Every feature is associated with a layout version, and an API corresponding > to the feature cannot be invoked (throws NOT_SUPPORTED_OPERATION) before > finalization. > * Created an annotation based 'aspect' for "guarding" new APIs that are > introduced by Layout Features. Check out > TestOMLayoutFeatureAspect#testCheckLayoutFeature. > * Added sample features and tests for ease of review (To be removed before > commit). > * Created an abstract VersionManager and an inherited OM Version manager to > initialize features, check if feature is allowed, check need to finalize, do > finalization. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4150) recon.api.TestEndpoints is flaky
[ https://issues.apache.org/jira/browse/HDDS-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4150: Fix Version/s: (was: 0.7.0) > recon.api.TestEndpoints is flaky > > > Key: HDDS-4150 > URL: https://issues.apache.org/jira/browse/HDDS-4150 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Marton Elek >Assignee: Vivek Ratnavel Subramanian >Priority: Blocker > > Failed on the PR: > https://github.com/apache/hadoop-ozone/pull/1349 > And on the master: > https://github.com/elek/ozone-build-results/blob/master/2020/08/25/2533/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt > and here: > https://github.com/elek/ozone-build-results/blob/master/2020/08/22/2499/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4150) recon.api.TestEndpoints is flaky
[ https://issues.apache.org/jira/browse/HDDS-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4150: Affects Version/s: 0.7.0 > recon.api.TestEndpoints is flaky > > > Key: HDDS-4150 > URL: https://issues.apache.org/jira/browse/HDDS-4150 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Marton Elek >Assignee: Vivek Ratnavel Subramanian >Priority: Blocker > Fix For: 0.7.0 > > > Failed on the PR: > https://github.com/apache/hadoop-ozone/pull/1349 > And on the master: > https://github.com/elek/ozone-build-results/blob/master/2020/08/25/2533/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt > and here: > https://github.com/elek/ozone-build-results/blob/master/2020/08/22/2499/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4150) recon.api.TestEndpoints is flaky
[ https://issues.apache.org/jira/browse/HDDS-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4150: Fix Version/s: 0.7.0 > recon.api.TestEndpoints is flaky > > > Key: HDDS-4150 > URL: https://issues.apache.org/jira/browse/HDDS-4150 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Marton Elek >Assignee: Vivek Ratnavel Subramanian >Priority: Blocker > Fix For: 0.7.0 > > > Failed on the PR: > https://github.com/apache/hadoop-ozone/pull/1349 > And on the master: > https://github.com/elek/ozone-build-results/blob/master/2020/08/25/2533/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt > and here: > https://github.com/elek/ozone-build-results/blob/master/2020/08/22/2499/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4150) recon.api.TestEndpoints is flaky
[ https://issues.apache.org/jira/browse/HDDS-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4150: --- Assignee: Vivek Ratnavel Subramanian > recon.api.TestEndpoints is flaky > > > Key: HDDS-4150 > URL: https://issues.apache.org/jira/browse/HDDS-4150 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Marton Elek >Assignee: Vivek Ratnavel Subramanian >Priority: Blocker > > Failed on the PR: > https://github.com/apache/hadoop-ozone/pull/1349 > And on the master: > https://github.com/elek/ozone-build-results/blob/master/2020/08/25/2533/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt > and here: > https://github.com/elek/ozone-build-results/blob/master/2020/08/22/2499/unit/hadoop-ozone/recon/org.apache.hadoop.ozone.recon.api.TestEndpoints.txt -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3881) Add current layout version to OM Ratis Request
[ https://issues.apache.org/jira/browse/HDDS-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-3881: --- Assignee: Aravindan Vijayan (was: Prashant Pogde) > Add current layout version to OM Ratis Request > -- > > Key: HDDS-3881 > URL: https://issues.apache.org/jira/browse/HDDS-3881 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > > To make sure the correct version of the applyTxn step is executed against the > request, we should add the version to the OM Request and use that version in > the applyTxn step. Add the current layout version (MLV) to the OM Ratis > request. If there is no layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-4143) Implement a version factory for OM Apply Transaction that uses the implementation based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4143: --- Assignee: Aravindan Vijayan (was: Prashant Pogde) > Implement a version factory for OM Apply Transaction that uses the > implementation based on layout version. > -- > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Fix For: 0.7.0 > > > Add the current layout version (MLV) to the OM Ratis request. If there is no > layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3698) Ozone Non-Rolling upgrades.
[ https://issues.apache.org/jira/browse/HDDS-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3698: Attachment: Ozone Non-Rolling Upgrades Doc v1.1.pdf > Ozone Non-Rolling upgrades. > --- > > Key: HDDS-3698 > URL: https://issues.apache.org/jira/browse/HDDS-3698 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Attachments: Ozone Non-Rolling Upgrades (Presentation).pdf, Ozone > Non-Rolling Upgrades Doc v1.1.pdf, Ozone Non-Rolling Upgrades.pdf > > > Support for Non-rolling upgrades in Ozone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3830) Introduce OM layout version 'v0'.
[ https://issues.apache.org/jira/browse/HDDS-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-3830: --- Assignee: Stephen O'Donnell (was: Aravindan Vijayan) > Introduce OM layout version 'v0'. > - > > Key: HDDS-3830 > URL: https://issues.apache.org/jira/browse/HDDS-3830 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Stephen O'Donnell >Priority: Major > Labels: upgrade-p0 > > The first layout version for OzoneManager will be '0' which will be written > to the version file. Until a future Ozone release with Upgrade & Finalize > support, this will just be a dummy number, to support backward compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a version factory for OM Apply Transaction that uses the implementation based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Summary: Implement a version factory for OM Apply Transaction that uses the implementation based on layout version. (was: Implement a ) > Implement a version factory for OM Apply Transaction that uses the > implementation based on layout version. > -- > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Fix For: 0.7.0 > > > Add the current layout version (MLV) to the OM Ratis request. If there is no > layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a version factory for OM Apply Transaction that uses the implementation based on layout version.
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Parent: HDDS-3698 Issue Type: Sub-task (was: Task) > Implement a version factory for OM Apply Transaction that uses the > implementation based on layout version. > -- > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Fix For: 0.7.0 > > > Add the current layout version (MLV) to the OM Ratis request. If there is no > layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-3881) Add current layout version to OM Ratis Request.
[ https://issues.apache.org/jira/browse/HDDS-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-3881: Description: To make sure the correct version of the applyTxn step is executed against the request, we should add the version to the OM Request and use that version in the applyTxn step. Add the current layout version (MLV) to the OM Ratis request. If there is no layout version present, we can default to '0'. (was: To make sure the correct version of the applyTxn step is executed against the request, we should add the version to the OM Request and use that version in the applyTxn step. ) > Add current layout version to OM Ratis Request. > --- > > Key: HDDS-3881 > URL: https://issues.apache.org/jira/browse/HDDS-3881 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Priority: Major > > To make sure the correct version of the applyTxn step is executed against the > request, we should add the version to the OM Request and use that version in > the applyTxn step. Add the current layout version (MLV) to the OM Ratis > request. If there is no layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4143) Implement a
[ https://issues.apache.org/jira/browse/HDDS-4143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4143: Summary: Implement a (was: Introduce version in OM Ratis request.) > Implement a > > > Key: HDDS-4143 > URL: https://issues.apache.org/jira/browse/HDDS-4143 > Project: Hadoop Distributed Data Store > Issue Type: Task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > Fix For: 0.7.0 > > > Add the current layout version (MLV) to the OM Ratis request. If there is no > layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-3881) Add current layout version to OM Ratis Request.
[ https://issues.apache.org/jira/browse/HDDS-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-3881: --- Assignee: Prashant Pogde > Add current layout version to OM Ratis Request. > --- > > Key: HDDS-3881 > URL: https://issues.apache.org/jira/browse/HDDS-3881 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Aravindan Vijayan >Assignee: Prashant Pogde >Priority: Major > > To make sure the correct version of the applyTxn step is executed against the > request, we should add the version to the OM Request and use that version in > the applyTxn step. Add the current layout version (MLV) to the OM Ratis > request. If there is no layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4143) Introduce version in OM Ratis request.
Aravindan Vijayan created HDDS-4143: --- Summary: Introduce version in OM Ratis request. Key: HDDS-4143 URL: https://issues.apache.org/jira/browse/HDDS-4143 Project: Hadoop Distributed Data Store Issue Type: Task Components: Ozone Manager Reporter: Aravindan Vijayan Assignee: Prashant Pogde Fix For: 0.7.0 Add the current layout version (MLV) to the OM Ratis request. If there is no layout version present, we can default to '0'. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4142) Expose upgrade related state through JMX
[ https://issues.apache.org/jira/browse/HDDS-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4142: Summary: Expose upgrade related state through JMX (was: Expose upgrade related state through CLI & JMX) > Expose upgrade related state through JMX > > > Key: HDDS-4142 > URL: https://issues.apache.org/jira/browse/HDDS-4142 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Istvan Fajth >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-4142) Expose upgrade related state through CLI & JMX
Aravindan Vijayan created HDDS-4142: --- Summary: Expose upgrade related state through CLI & JMX Key: HDDS-4142 URL: https://issues.apache.org/jira/browse/HDDS-4142 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Aravindan Vijayan Assignee: Istvan Fajth Fix For: 0.7.0 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-4142) Expose upgrade related state through CLI & JMX
[ https://issues.apache.org/jira/browse/HDDS-4142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-4142: Parent: HDDS-3698 Issue Type: Sub-task (was: Bug) > Expose upgrade related state through CLI & JMX > -- > > Key: HDDS-4142 > URL: https://issues.apache.org/jira/browse/HDDS-4142 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Aravindan Vijayan >Assignee: Istvan Fajth >Priority: Major > Fix For: 0.7.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
