[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122790#comment-16122790 ] Weiwei Yang commented on HDFS-12196: Hi [~anu] Thanks for your quick response. I will address these comments in this jira with next patch, they are not big changes so lets address them here. Thank you. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122786#comment-16122786
]
Anu Engineer commented on HDFS-12196:
-
[~cheersyang] +1 for v5 patch, pending Jenkins.
I have a small nit, which *we don't need to address* in this patch. My
apologies that I did not spot it earlier.
Instead of the word *recycling* can we use the word *delete*? I feel that it is
easier to understand.
Rename ContainerRecyclingService to something like BlockDeletingService.
Also, change the comment for this class to something like:
{noformat}
A per-datanode block deleting service that deletes
blocks from active containers.
{noformat}
You don't have to do this now, please feel free to commit. I know you have 3
more checkins pending on this, so feel free to modify this name is some later
patch.
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch,
> HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch,
> HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122780#comment-16122780
]
Weiwei Yang commented on HDFS-12196:
Thanks [~anu] for the comments. Per offline discussion, since we will reuse
background service for multiple components, KSM, SCM as well as datanode, lets
use separate thread pools. For rest of your comments
bq. Generally we expect the background service to be a deamon thread.
Fixed. Now it is constructed like following way
{code}
ThreadFactory tf = r -> new Thread(threadGroup, r);
threadFactory = new ThreadFactoryBuilder()
.setThreadFactory(tf)
.setDaemon(true)
.setNameFormat(serviceName + "#%d")
.build();
{code}
this is to ensure we have all threads contained in a thread group, to manage
them all together. So we can get # of running threads for this service if
necessary.
bq. Question : Why do we need the testing flag and testing Thread?
I have refactor that to a single-test-purpose class
{{ContainerRecyclingServicetTestImpl}}, instead of waiting for intervals, the
test class run each cycle by a function call, so that I can write more
finer-grained UT cases.
bq. BackgroundTaskQueue.java is not thread safe, is it by design?
This class doesn't have to be thread safe as there is only 1 thread to fetch
the tasks. But I use a thread safe implementation in the new patch, in case in
future we need to access it in multiple threads.
bq. Should we create a new package called background tasks ...
Fixed.
Thank you!
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch,
> HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch,
> HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122449#comment-16122449
]
Anu Engineer commented on HDFS-12196:
-
[~cheersyang] , Thanks for updating the patch. It looks very good. I am almost
a +1 for this change. I had some minor questions/comments.
h6. One Design question
Should we allow each background task to create its own thread pool or just have
a common thread pool?
In that case, we will need to have a single service called BackgroundService --
and all other jobs like "ContainerRecyclingService" will become a task, say
"ContainerRecyclingTask".
In other words, I am saying that we can just queue "ContainerRecyclingTask"
directly to a background service without having individual job execution pools.
Just one single common pool for all background jobs.
Just wondering if that is an interface we want to offer. I do see the down
side, we can have a task monopolize the thread pool. If you are free ping me in
slack or we can have an offline chat about this.
I am also open to checking in this architecture and may be refining it later.
h6. some code comments
* {{BackgroundService.java}}
bq. threadFactory = r -> new Thread(threadGroup, r);
Generally we expect the background service to be a
deamon thread. Would you like to replace this with
something like
{code}
new ThreadFactoryBuilder().setDaemon(true)
.setNameFormat( threadName + "#%d")
.build());
{code}
Where threadName is an argument to the ctor ?
* {{BackgroundService.java}}
Question : Why do we need the testing flag and testing Thread?
* {{BackgroundTaskQueue.java}}
Is not thread safe, is it by design? Just wondering if it is possible for 2
different threads to call into this concurrently. From a quick code reading, I
am not able to see it, may we can just add a comment to the class.
* {{ContainerRecyclingService}}
Should we create a new package called *background* tasks under
{{org.apache.hadoop.ozone.container.common.statemachine}}
I am presuming we will need much more tasks like this in future.
Ps. Sorry for the delay in code review, I was focused on the pluggable pipeline
patch.
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch,
> HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch,
> HDFS-12196-HDFS-7240.004.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119513#comment-16119513
]
Hadoop QA commented on HDFS-12196:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 40s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}112m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.scm.TestArchive |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.cblock.TestCBlockReadWrite |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880962/HDFS-12196-HDFS-7240.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 7e6a0ef1bb44 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 43d3811 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119341#comment-16119341 ] Anu Engineer commented on HDFS-12196: - [~cheersyang] Sorry for the delay. I will get to this latest by tomorrow evening. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119332#comment-16119332 ] Weiwei Yang commented on HDFS-12196: Hi [~anu], [~vagarychen], please let me know your thoughts to the v3 patch. I am working on DeleteKey-3 now, which is a little bit depending on this one. Thank you. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116400#comment-16116400
]
Hadoop QA commented on HDFS-12196:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 96m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.cblock.TestBufferManager |
| | hadoop.ozone.web.client.TestKeys |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880610/HDFS-12196-HDFS-7240.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 09e7cc8cc12f 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / b153dbb |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20578/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20578/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116274#comment-16116274 ] Weiwei Yang commented on HDFS-12196: Hi [~vagarychen] Thanks for raising up your concern. The reason I did not use LRUMap or Guava cache here because they both evict something when cache gets full. But in this case, I don't want to remove any thing from cache (become an incomplete result set) so I claim it as a fixed size list, once reaches its cap then pause further task executions. This might be a bit over designed at present, given the truth we are not going to use the cache any time soon, I removed that part in v3 patch. Lets get the basic stuff working first. If moving on we need to implement anything similar, we can revisit this. Thanks for sharing your thought, it is quite helpful. I have uploaded v3 patch. Please help to review. Thanks. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113492#comment-16113492
]
Chen Liang commented on HDFS-12196:
---
Thanks [~cheersyang] for the reply and the updated patch!
I'm still concerned about the result caching though...I agree with you that it
can save RPC calls, but the thing is it can be very tricky to get the caching
work properly...more specifically:
1. One potential issue is that it seems there is no purging of the cache. i.e.
once an entry gets added to the {{resultList}}, it never gets removed. So the
size of {{resultList}} will monolithically increase to the point where no more
entries can be added and then the result cache will no longer be helpful for
any further entry. Also since all entries are just being added when there is
space, if there is no more query for those cached result, without purging, we
may end up just holding a bunch of objects in memory that will never be useful.
I think ideally the cache should probably be removing entries based on factors
such as access time, frequency etc. In short, the cache needs to have purging.
2. Seems there is no easy way to check the current {{resultList}} efficiently.
Any service using it will have to check the entire cache. Namely, since it is a
list, so even if some service wants to check if the result of a call is cached
in it before making the call, it still needs to iterate over all the entries in
it, so if the list is large enough, it could be even less efficient than
contacting datanodes. In short, the cache needs to have fast lookup. Another
side thing is that the entire {{resultList}} object is exposed and returned to
caller, a bug in caller can easily mess up the list...
I'm not against adding cache here, just saying we should probably pay a little
attention. If we really want cache here, it shouldn't be hard to implement a
cache on top of either
[LRUMap|https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/map/LRUMap.html]
OR [Guava loading cache|https://github.com/google/guava/wiki/CachesExplained],
which should make purging and fast lookup very easy, even trivial. (One example
is the cache in {{XceiverClientManager}}, just FYI).
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch,
> HDFS-12196-HDFS-7240.002.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108712#comment-16108712
]
Hadoop QA commented on HDFS-12196:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
1 unchanged - 0 fixed = 8 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.ksm.TestKSMMetrcis |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879793/HDFS-12196-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 3499c2c907d8 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 1fd2790 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/20516/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108607#comment-16108607
]
Weiwei Yang commented on HDFS-12196:
[~vagarychen]
thanks for the review.
bq. Can we somehow add the priority enforcement to the abstractions also?
It makes sense, I have added a {{BackgroundTaskQueue}} for better encapsulation.
bq. I was wondering will there be anyone that will be reading RecyclingResult?
I suppose so. It is not required now, but good to have. For example with cached
result, DN could answer back SCM deleted block list without querying its
database. And as now it becomes to be a more general interface, it might be
useful in other scenarios.
bq. Also I felt the term "recycling" here is not that informative, because it
does not seem that we recycling anything, but can't think of a better
alternative...
the name of "recycling" implies the service is trying to delete staled stuff
and recycle space for ozone. But just let me know if you have any better idea
about the naming.
[~anu]
Thanks for your review. I have created {{BackgroundService}},
{{BackgroundTask}}, do you think this is general enough for ozone background
services? Please take a look at v2 patch.
Thank you.
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch,
> HDFS-12196-HDFS-7240.002.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108138#comment-16108138 ] Anu Engineer commented on HDFS-12196: - bq. That absolutely makes sense. I have already created an abstract class AbstractRecyclingService which is a base class for all Recycling Services I was suggesting something little more ambitious. I was suggesting that we need to create an interface called Background tasks for KSM, SCM and Datanode. All tasks like Recycling task or volume scanner or garbage collection tasks for incomplete objects etc, Should run in that mode. I do see that having a Recycling Interface is nice, so it is easy to see how KSM Recycling, SCM Recycling, and Datanode stuff all relate to each other. I would really love if they are all part of Background Task interface for Ozone. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108029#comment-16108029
]
Chen Liang commented on HDFS-12196:
---
Thanks [~cheersyang] for the patch, looks pretty good to me overall! Some
initial comments:
1. Although {{RecyclingTask}} has a {{getPriority()}} method, it seems that
only the implementation class of {{AbstractRecyclingService}} will be enforcing
the priority (currently, {{ContainerRecyclingService}}). So if more
implementation of AbstractRecyclingService get added later, each of them will
need to have this line
{code}
PriorityQueue tasks = new PriorityQueue<>(
(task1, task2) -> task1.getPriority() - task2.getPriority());
{code}
Otherwise seems the priorities will be completely ignored? Can we somehow add
the priority enforcement to the abstractions also? e.g. somehow add a abstract
priority queue class or something.
2. I was wondering will there be anyone that will be reading
{{RecyclingResult}}? In other words, will there be cases that some logic will
check the result of recycling tasks and take action accordingly? Seems unit
test is the only one that does this in this patch. If not, then having this
abstraction only to make unit test work seems a little bit of a overkill to
me...This is also related to the result caching part.
Also I felt the term "recycling" here is not that informative, because it does
not seem that we recycling anything, but can't think of a better alternative...
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105945#comment-16105945
]
Weiwei Yang commented on HDFS-12196:
Hi [~anu]
Thanks for the quick response
bq. Should we have generic background tasks Service?
That absolutely makes sense. I have already created an abstract class
{{AbstractRecyclingService}} which is a base class for all Recycling Services,
so moving on there will be {{KsmRecyclingService}}, {{ScmRecyclingService}} and
{{ContainerRecyclingService}}. What you are suggesting sounds like we need a
parent for all these backend services, but need to think about the base class
design to let it reusable for them. I will think about this.
bq. If we move to that architecture, then we should have a queue on to which
the commands are queued in the data node.
That will be the work in next task. This patch only contains an independent
recycling service, scans containers db in certain interval and deletes stuff.
In next task, I will create a {{BlockDeletionCommandHandler}} which hands the
block deletion command sent by SCM, that handler will just move blocks to
deleting state in containers db. This work will be done with SCM->Datanode
interaction task.
Let me know if you have more thoughts, I can make improvements in v2 patch then.
Thank you.
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105438#comment-16105438
]
Hadoop QA commented on HDFS-12196:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 7 new +
1 unchanged - 0 fixed = 8 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879371/HDFS-12196-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux af773aaa3f94 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105350#comment-16105350 ] Anu Engineer commented on HDFS-12196: - Hi [~cheersyang], Thanks for posting the first patch. Some initial thoughts. I am still going thru the patch. 1. Should we have generic background tasks Service? There will be many background tasks that we might have to run, the delete thread, volumeScanner, container DB compaction task, Container Compress Task, CopyContainer Task etc. etc. Given that we have a list of background task we would like to run, should recycle service or the recycle task be one of that task? In other words, I am proposing the we prompt your recycleService to be more generic background task for datanode. 2. If we move to that architecture, then we should have a queue on to which the commands are queued in the data node. This task service can pick up the commands and execute the correct handler. We already have a command handler interface in state machine classes. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[
https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105257#comment-16105257
]
Weiwei Yang commented on HDFS-12196:
Uploaded v1 patch, added a {{ContainerRecyclingService}}
It run a container recycling service per datanode, it runs in a configurable
time interval (default 1 minute), in each interval, it scans {{staled}} blocks
(blocks with #deleting# prefix) in container's db and delete the associated
chunks from disk. If all succeed, remove the blocks from db.
Some other features of this service
* The recycling services spawns a thread per container, property
{{ozone.recycling.container.limit.per.interval}} is used to throttle the number
of containers to process at a time.
* Each container thread will delete a certain number of blocks at a time, this
is throttled by property {{ozone.recycling.block.limit.per.task}}.
* There is a general interface {{RecyclingResult}} that can be used to collect
recycling task result. And the result can be cached in case some other place
needs to play with them. If cache is enabled, there is a size limit. Once the
limit is reached, the recycling service will pause itself from new work until
some of the results in the cache get released.
cc [~anu], [~xyao], please kindly review. Thanks!
> Ozone: DeleteKey-2: Implement container recycling service to delete stale
> blocks at background
> --
>
> Key: HDFS-12196
> URL: https://issues.apache.org/jira/browse/HDFS-12196
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-12196-HDFS-7240.001.patch
>
>
> Implement a recycling service running on datanode to delete stale blocks.
> The recycling service scans staled blocks for each container and delete
> chunks and references periodically.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
