[
https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582375#comment-16582375
]
Elek, Marton commented on HDDS-328:
-----------------------------------
Thank you [~hanishakoneru] and [~xyao] the review comments. In the latest patch
I addressed all of them. Especially:
1. I modified the interface of the ContainerPacker. During the compression it
compresses all the data (conainer yaml + db + chunks). For the decompression we
have two methods: one to unpack db + chunks and one to read the container yaml.
The data unpack extracts just the db and the chunks but returns with the
content of the container yaml instead of writing it out. With this method the
container file will be written only at the end of the import. (your 2nd method).
2. I added a check about the existence of the container yaml file. If it exists
the container import will be failed. (I will check the existence of the
container in the ContainerSet as part of HDDS-75. In the ContainerData.import I
have no information about the current set).
3. I delete all the chunk/metadata/yaml file in case of an unsuccessful
import.
4. [~xyao] bzip or lz4: Yes, that's the reason for the ContainerPacker
interface. If you have suggestion for faster/smaller format I can create
multiple (newbie?) jira-s to implement them: 1.) make the container packer
configurable 2) provider different implementation (which one? tar+bzip? tar +
lz?)
5. [~xyao] "Line 81/91: we should avoid using hard-coded path separators the
applies only to *nix OS."
To be honest I was not aware of this problem. But after checking it, it seems
to be fine. According to the
[SO|https://stackoverflow.com/questions/31600127/does-os-path-sep-affect-the-tarfile-module]
tar archives always use the slash *inside* the archive. These slash usages
only for the path names inside the container. For the external (real path) I
used path.resolve.
6. EOF handling is fixed (EOF should not be possible here, but I added a check,
and set remaining to zero if read returns with -1)
7. try-with-resource: added for both the input and the output files.
8. Duplicated code is removed.
Thank you again, your review.
> Support export and import of the KeyValueContainer
> --------------------------------------------------
>
> Key: HDDS-328
> URL: https://issues.apache.org/jira/browse/HDDS-328
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Elek, Marton
> Assignee: Elek, Marton
> Priority: Critical
> Fix For: 0.2.1
>
> Attachments: HDDS-328.002.patch, HDDS-328.003.patch,
> HDDS-328.004.patch
>
>
> In HDDS-75 we pack the container data to an archive file, copy to other
> datanodes and create the container from the archive.
> As I wrote in the comment of HDDS-75 I propose to separate the patch to make
> it easier to review.
> In this patch we need to extend the existing Container interface with adding
> export/import methods to save the container data to one binary input/output
> stream.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
