[
https://issues.apache.org/jira/browse/HDDS-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110194#comment-17110194
]
Marton Elek edited comment on HDDS-3499 at 5/21/20, 8:35 AM:
-------------------------------------------------------------
I tested the upgrade today, and I upgraded a docker-compose based cluster
without any problem. Here are the steps what I followed:
0. I created a build from the revision (0466ade7d~1) to use the last version
before HDDS-3172.
1. I started a docker-compose cluster but I mounted the SCM metadata to an
external dir, to make it available during the test:
(See the ./scmdata in the docker-compose)
{code}
scm:
<<: *common-config
ports:
- 9876:9876
volumes:
- ./scmdata:/data
- ../..:/opt/hadoop
{code}
2. Started the ozone cluster, executed `ozone freon ockg -n10` and uploaded a
key manually.
3. I compiled the ldb binary from the source
https://github.com/facebook/rocksdb/tree/master/tools based on the rocksdb
documentation
4. I stopped the SCM.
5. I created the two missing column family in the new db:
{code}
ldb --db=`pwd`/scm.db create_column_family pipelines
ldb --db=`pwd`/scm.db create_column_family containers
{code}
6. I dumped the old data:
{code}
ldb --db=`pwd`/scm-container.db --key_hex --value_hex dump > /tmp/container.db
ldb --db=`pwd`/scm-pipeline.db --key_hex --value_hex dump > /tmp/pipeline.db
{code}
5. And loaded them to the existing scm.db:
{code}
cat /tmp/container.db | ~/projects/ldb --db=`pwd`/scm.db --key_hex --value_hex
--column_family=containers load
cat /tmp/pipeline.db | ~/projects/ldb --db=`pwd`/scm.db --key_hex --value_hex
--column_family=pipelines load
{code}
6. Restarted the SCM with mounting the new build from latest master (stored in
../../../after in my case)
{code}
scm:
<<: *common-config
ports:
- 9876:9876
volumes:
- ./scmdata:/data
- ../../../after:/opt/hadoop
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode:
${OZONE_SAFEMODE_MIN_DATANODES:-1}
<<: *replication
command: ["ozone","scm"]
{code}
7. Tested with `ozone admin container list`, and with downloading my earlier
file (it was downloadable)
was (Author: elek):
I tested the upgrade today, and I upgraded a docker-compose based cluster
without any problem. Here are the steps what I followed:
0. I created a build from the revision (0466ade7d~1) to use the last version
before HDDS-3172.
1. I started a docker-compose cluster but I mounted the SCM metadata to an
external dir, to make it available during the test:
(See the ./scmdata in the docker-compose)
{code}
scm:
<<: *common-config
ports:
- 9876:9876
volumes:
- ./scmdata:/data
- ../..:/opt/hadoop
{code}
2. Started the ozone cluster, executed `ozone freon ockg -n10` and uploaded a
key manually.
3. I compiled the ldb binary from the source
https://github.com/facebook/rocksdb/tree/master/tools based on the rocksdb
documentation
4. I stopped the SCM and started the updated process.
5. I created the two missing column family in the new db:
{code}
ldb --db=`pwd`/scm.db create_column_family pipelines
ldb --db=`pwd`/scm.db create_column_family containers
{code}
6. I dumped the old data:
{code}
ldb --db=`pwd`/scm-container.db --key_hex --value_hex dump > /tmp/container.db
ldb --db=`pwd`/scm-pipeline.db --key_hex --value_hex dump > /tmp/pipeline.db
{code}
5. And loaded them to the existing scm.db:
{code}
cat /tmp/container.db | ~/projects/ldb --db=`pwd`/scm.db --key_hex --value_hex
--column_family=containers load
cat /tmp/pipeline.db | ~/projects/ldb --db=`pwd`/scm.db --key_hex --value_hex
--column_family=pipelines load
{code}
6. Restarted the SCM with mounting the new build from latest master (stored in
../../../after in my case)
{code}
scm:
<<: *common-config
ports:
- 9876:9876
volumes:
- ./scmdata:/data
- ../../../after:/opt/hadoop
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode:
${OZONE_SAFEMODE_MIN_DATANODES:-1}
<<: *replication
command: ["ozone","scm"]
{code}
7. Tested with `ozone admin container list`, and with downloading my earlier
file (it was downloadable)
> Address compatibility issue by SCM DB instances change
> ------------------------------------------------------
>
> Key: HDDS-3499
> URL: https://issues.apache.org/jira/browse/HDDS-3499
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Li Cheng
> Assignee: Marton Elek
> Priority: Blocker
>
> After https://issues.apache.org/jira/browse/HDDS-3172, SCM now has one single
> rocksdb instance instead of multiple db instances.
> For running Ozone cluster, we need to address compatibility issues. One
> possible way is to have a side-way tool to migrate old metadata from multiple
> dbs to current single db.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
