[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393517#comment-17393517
]
liuyongpan commented on HDFS-16147:
---
{color:#33}I am very glad to receive your reply:D. I'll take your advice. I
have tested this patch on a large image, see attached figure.
{color}
> load fsimage with parallelization and compression
> -
>
> Key: HDFS-16147
> URL: https://issues.apache.org/jira/browse/HDFS-16147
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
>Affects Versions: 3.3.0
>Reporter: liuyongpan
>Priority: Minor
> Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch, compress_not
> parallel.png, compress_parallel.png, subsection.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393386#comment-17393386
]
Stephen O'Donnell commented on HDFS-16147:
--
With your patch in place I think the output file looks like this:
{code}
OVERALL_STREAM
INODE_SECTION (only in index, not in the data stream)
COMPRESSED_INODE_SUB_SECTION
COMPRESSED_INODE_SUB_SECTION
COMPRESSED_INODE_SUB_SECTION
...
EMPTY_COMPRESSED_INODE_SUB_SECTION
...
DIR_SECTION (only in index, not in the data stream)
COMPRESSED_DIR_SUB_SECTION
COMPRESSED_DIR_SUB_SECTION
...
EMPTY_COMPRESSED_DIR_SUB_SECTION
...
{code}
The reason for the empty ones, is because at the end of the INODE section, you
call commitSectionAndSubSection() and it closes the sub-section and opens a new
compressed stream. Then you immediately close the section, which closes it and
opens a new one. I don't think it does any harm, but it would be better if it
did not do that, if we can fix it without making the code too complex.
Then I think the reason this works, is that if you try to read in parallel, it
reads each compressed sub-section. This is fine.
When you try to read it serially (ie turn parallel off and load an image or use
OIV), it will try to read all the compressed INODE sections using a single
stream. I think this is like a series of streams concatenated together, and the
decompressor must handle that (concatenated streams) and return the output like
it is a single compressed stream. We can probably test this out somehow to be
sure.
In TestFSImage.testNoParallelSectionsWithCompressionEnabled(..) - could you
remove or rename that test and make it load an image with parallel enabled
rather than the current test which checks it does not work.
Provided my understanding is correct, this patch looks mostly good apart from
the two things above, but I will give it a more detailed review in the next day
or two.
Have you tested this patch on a large image with millions of inodes?
> load fsimage with parallelization and compression
> -
>
> Key: HDFS-16147
> URL: https://issues.apache.org/jira/browse/HDFS-16147
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
>Affects Versions: 3.3.0
>Reporter: liuyongpan
>Priority: Minor
> Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch,
> subsection.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393356#comment-17393356
]
Stephen O'Donnell commented on HDFS-16147:
--
On further review, most of what I wrote above is wrong!
When saving the image, there is a single output stream, but each section is
compressed within that stream, each as a separate compressed stream, eg:
{code}
OVERALL_STREAM
COMPRESSED_INODE_SECTION
COMPRESSED_DIR_SECTION
...
{code}
You can see this in the commitSection() method, where the stream is finished().
So this means that when we load a section (not in parallel), it jumps to the
start of a compressed section, and reads it in full.
This means it is still unknown how you can save a compressed image with
sub-sections and load it without parallel. Perhaps a compressed stream can read
embedded compressed streams within itself - I am not sure, but I would like to
understand how this is working.
> load fsimage with parallelization and compression
> -
>
> Key: HDFS-16147
> URL: https://issues.apache.org/jira/browse/HDFS-16147
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
>Affects Versions: 3.3.0
>Reporter: liuyongpan
>Priority: Minor
> Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch,
> subsection.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393310#comment-17393310
]
Stephen O'Donnell commented on HDFS-16147:
--
When the image is saved, there is a single stream written out serially. To
enable parallel load on the image, index entries are added for the sub-sections
as they are written out.
This means we have a single stream, with the position of the sub-sections saved.
That means, when we load the image, there are two choices:
1. We start at the beginning of a section and open a stream and read the entire
section.
2. We open several streams at by jumping to that position and read the given
length.
When you enabled compression too, this means the entire stream is compressed
from end to end as a single compressed stream. I wrongly thought there would be
many compressed streams within the image file, and that is why I though OIV etc
would have trouble reading this.
So it makes sense OIV can read the image serially, and the namenode can also
read the image with parallel disabled when compression is on. The surprise to
me, is that we can load the image in parallel, as that involves jumping into
the compressed stream somewhere in the middle and starting to read, which more
compression codecs do not support. It was my belief that gzip does not support
this.
However, looking at the existing code, before this change, I see that we jump
around in the stream already:
{code}
for (FileSummary.Section s : sections) {
channel.position(s.getOffset());
InputStream in = new BufferedInputStream(new LimitInputStream(fin,
s.getLength()));
in = FSImageUtil.wrapInputStreamForCompression(conf, summary.getCodec(), in);
{code}
So that must mean the compression codecs are splittable somehow, and they are
start decompressing from an random position in the stream. Due to this, if the
image is compressed, the existing parallel code can be mostly reused to load
the sub-sections within the compressed stream.
>From the above, could we allow parallel loading of compressed images by simply
>removing the code:
{code}
-if (loadInParallel) {
- if (compressionEnabled) {
-LOG.warn("Parallel Image loading and saving is not supported when {}" +
-" is set to true. Parallel will be disabled.",
-DFSConfigKeys.DFS_IMAGE_COMPRESS_KEY);
-loadInParallel = false;
- }
-}
{code}
Then let the image save compressed with the sub-sections indexed and try to
load it?
> load fsimage with parallelization and compression
> -
>
> Key: HDFS-16147
> URL: https://issues.apache.org/jira/browse/HDFS-16147
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
>Affects Versions: 3.3.0
>Reporter: liuyongpan
>Priority: Minor
> Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch,
> subsection.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391605#comment-17391605 ] liuyongpan commented on HDFS-16147: --- 1、HDFS-16147.002.patch fix the error of test unit at org.apache.hadoop.hdfs.server.namenode.TestFSImage.testNoParallelSectionsWithCompressionEnabled 2、Upon careful examination, Oiv can indeed work normally, and I can't explain why it works. 3、If I create a parallel compressed image with this patch, and then try to load it in a NN without this patch and parallel loading disabled, the NN still able to load it. It may be hard to understand, but it's true, and you can check it out. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch, HDFS-16147.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390089#comment-17390089
]
Hadoop QA commented on HDFS-16147:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
46s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 17s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m
7s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 3m
21s{color} | {color:green}{color} |
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389947#comment-17389947 ] liuyongpan commented on HDFS-16147: --- [~sodonnell] ,thank you very much for your advice. I will check it carefully. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[
https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389923#comment-17389923
]
Stephen O'Donnell commented on HDFS-16147:
--
It is important that OIV can read these images.
If you create a parallel compressed image with this patch, and then try to load
it in a NN without this patch and parallel loading disabled, is the NN still
able to load it? I suspect it won't as there are multiple compressed sections,
so it cannot read a single compressed stream from end to end.
{quote}
It can load 300M Fsimage with compression and parallelization , which can
improve 50% loading time.
{quote}
Is the 50% improvement measured against a compressed single threaded load
verses parallel compressed loading?
How are the load times between "parallel not compressed " vs "parallel
compressed"?
> load fsimage with parallelization and compression
> -
>
> Key: HDFS-16147
> URL: https://issues.apache.org/jira/browse/HDFS-16147
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
>Affects Versions: 3.3.0
>Reporter: liuyongpan
>Priority: Minor
> Attachments: HDFS-16147.001.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389905#comment-17389905 ] liuyongpan commented on HDFS-16147: --- sorry , OIV can't work > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389888#comment-17389888 ] liuyongpan commented on HDFS-16147: --- [~sodonnell] I tested it in my test environment，and checked your question. 1、OIV can read the image, but can not read the image in parallel, I will do it later. 2、Current logic can read both the old Fsimage and the new Fsimage, I have tested. 3、It can load 300M Fsimage with compression and parallelization , which can improve 50% loading time. > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16147) load fsimage with parallelization and compression
[ https://issues.apache.org/jira/browse/HDFS-16147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389880#comment-17389880 ] Stephen O'Donnell commented on HDFS-16147: -- I only quickly looked at this, but I have a few questions. With this change, will tools like OIV be able to read the image? It looks like there are a series of new compressed sections, and OIV currently does not read the image in parallel - it will just try to read it from the start to the end - will this work? If we have parallel enabled and compressed sub-sections, if we disable parallel for some reason, will the image be readable? Have you been able to benchmark loading a large image in parallel with compression enabled and disabled so we can see if compression makes it faster or slower? > load fsimage with parallelization and compression > - > > Key: HDFS-16147 > URL: https://issues.apache.org/jira/browse/HDFS-16147 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namanode >Affects Versions: 3.3.0 >Reporter: liuyongpan >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-16147.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
