[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176057#comment-16176057 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/133 [![Coverage Status](https://coveralls.io/builds/13384852/badge)](https://coveralls.io/builds/13384852) Coverage increased (+0.3%) to 38.964% when pulling **ba0348b972cbab3f7c4f296039c85c27c5c1c1e2 on shroman:ROCKETMQ-249** into **845830865fc37d0364a19cbd89ceaf8a30b37e1c on apache:develop**. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176040#comment-16176040 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user shroman commented on a diff in the pull request: https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140425644 --- Diff: store/src/test/java/org/apache/rocketmq/store/CleanCommitLogServiceTest.java --- @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.rocketmq.store; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import org.apache.rocketmq.common.BrokerConfig; +import org.apache.rocketmq.store.config.MessageStoreConfig; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + +/** + * Tests for CleanCommitLogService. + */ +public class CleanCommitLogServiceTest { +DefaultMessageStore messageStore; +DefaultMessageStore.CleanCommitLogService cleanCommitLogService; + +// CleanCommitLogService class. +Class clazz; + +@Before +public void setUp() throws Exception { +clazz = Class.forName("org.apache.rocketmq.store.DefaultMessageStore$CleanCommitLogService"); + +MessageStoreConfig messageStoreConfig = new MessageStoreConfig(); +messageStore = new DefaultMessageStore( +messageStoreConfig, null, null, new BrokerConfig()); + +Field cleanCommitLogServiceField = messageStore.getClass().getDeclaredField("cleanCommitLogService"); +cleanCommitLogServiceField.setAccessible(true); + +cleanCommitLogService = spy((DefaultMessageStore.CleanCommitLogService) (cleanCommitLogServiceField.get(messageStore))); +} + +@After +public void tearDown() throws Exception { +} + +@Test +public void isSpaceToDeleteNoCommitLog() throws Exception { +Method m = clazz.getDeclaredMethod("isSpaceToDelete"); +m.setAccessible(true); + +Assert.assertFalse((boolean) m.invoke(cleanCommitLogService)); +} + +@Test +public void isSpaceToDeleteWithCommitLogNoQueue() throws Exception { +try { +messageStore.start(); +messageStore.load(); + +Method m = clazz.getDeclaredMethod("isSpaceToDelete"); +m.setAccessible(true); + +Assert.assertFalse((boolean) m.invoke(cleanCommitLogService)); +} finally { +messageStore.shutdown(); --- End diff -- Good point. thanks. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176039#comment-16176039 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user shroman commented on a diff in the pull request: https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140425575 --- Diff: store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java --- @@ -1509,65 +1514,62 @@ private boolean isTimeToDelete() { return false; } -private boolean isSpaceToDelete() { -double ratio = DefaultMessageStore.this.getMessageStoreConfig().getDiskMaxUsedSpaceRatio() / 100.0; +double getDiskUsageRatio() { +return UtilAll.getDiskPartitionSpaceUsedPercent( + DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog()); +} -cleanImmediately = false; +double getQueueSpace() { +return UtilAll.getDiskPartitionSpaceUsedPercent(StorePathConfigHelper + .getStorePathConsumeQueue(DefaultMessageStore.this.getMessageStoreConfig().getStorePathRootDir())); +} -{ -String storePathPhysic = DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog(); -double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic); -if (physicRatio > diskSpaceWarningLevelRatio) { -boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskFull(); -if (diskok) { -DefaultMessageStore.log.error("physic disk maybe full soon " + physicRatio + ", so mark disk full"); -} +/** + * Checks if cleaning on the disk is needed. + * + * @param usageRatio Usage ratio. + * @param allowedRatio Allowed ratio. + * @return True if cleaning is needed, otherwise false. + */ +private boolean needCleaning(double usageRatio, double allowedRatio) { +if (usageRatio == -1) --- End diff -- Thanks, fixed. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176037#comment-16176037 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user shroman commented on the issue: https://github.com/apache/incubator-rocketmq/pull/133 @dongeforever Modified as you suggested, thanks! > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175815#comment-16175815 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user dongeforever commented on a diff in the pull request: https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140402297 --- Diff: store/src/test/java/org/apache/rocketmq/store/CleanCommitLogServiceTest.java --- @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.rocketmq.store; + +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import org.apache.rocketmq.common.BrokerConfig; +import org.apache.rocketmq.store.config.MessageStoreConfig; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.when; + +/** + * Tests for CleanCommitLogService. + */ +public class CleanCommitLogServiceTest { +DefaultMessageStore messageStore; +DefaultMessageStore.CleanCommitLogService cleanCommitLogService; + +// CleanCommitLogService class. +Class clazz; + +@Before +public void setUp() throws Exception { +clazz = Class.forName("org.apache.rocketmq.store.DefaultMessageStore$CleanCommitLogService"); + +MessageStoreConfig messageStoreConfig = new MessageStoreConfig(); +messageStore = new DefaultMessageStore( +messageStoreConfig, null, null, new BrokerConfig()); + +Field cleanCommitLogServiceField = messageStore.getClass().getDeclaredField("cleanCommitLogService"); +cleanCommitLogServiceField.setAccessible(true); + +cleanCommitLogService = spy((DefaultMessageStore.CleanCommitLogService) (cleanCommitLogServiceField.get(messageStore))); +} + +@After +public void tearDown() throws Exception { +} + +@Test +public void isSpaceToDeleteNoCommitLog() throws Exception { +Method m = clazz.getDeclaredMethod("isSpaceToDelete"); +m.setAccessible(true); + +Assert.assertFalse((boolean) m.invoke(cleanCommitLogService)); +} + +@Test +public void isSpaceToDeleteWithCommitLogNoQueue() throws Exception { +try { +messageStore.start(); +messageStore.load(); + +Method m = clazz.getDeclaredMethod("isSpaceToDelete"); +m.setAccessible(true); + +Assert.assertFalse((boolean) m.invoke(cleanCommitLogService)); +} finally { +messageStore.shutdown(); --- End diff -- How about putting the finally block into teardown() ? The test framework will handle it. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175814#comment-16175814 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user dongeforever commented on a diff in the pull request: https://github.com/apache/incubator-rocketmq/pull/133#discussion_r140402195 --- Diff: store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java --- @@ -1509,65 +1514,62 @@ private boolean isTimeToDelete() { return false; } -private boolean isSpaceToDelete() { -double ratio = DefaultMessageStore.this.getMessageStoreConfig().getDiskMaxUsedSpaceRatio() / 100.0; +double getDiskUsageRatio() { +return UtilAll.getDiskPartitionSpaceUsedPercent( + DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog()); +} -cleanImmediately = false; +double getQueueSpace() { +return UtilAll.getDiskPartitionSpaceUsedPercent(StorePathConfigHelper + .getStorePathConsumeQueue(DefaultMessageStore.this.getMessageStoreConfig().getStorePathRootDir())); +} -{ -String storePathPhysic = DefaultMessageStore.this.getMessageStoreConfig().getStorePathCommitLog(); -double physicRatio = UtilAll.getDiskPartitionSpaceUsedPercent(storePathPhysic); -if (physicRatio > diskSpaceWarningLevelRatio) { -boolean diskok = DefaultMessageStore.this.runningFlags.getAndMakeDiskFull(); -if (diskok) { -DefaultMessageStore.log.error("physic disk maybe full soon " + physicRatio + ", so mark disk full"); -} +/** + * Checks if cleaning on the disk is needed. + * + * @param usageRatio Usage ratio. + * @param allowedRatio Allowed ratio. + * @return True if cleaning is needed, otherwise false. + */ +private boolean needCleaning(double usageRatio, double allowedRatio) { +if (usageRatio == -1) --- End diff -- It is not safe to use == for float value. In this case, usageRatio < 0 is OK > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125152#comment-16125152 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user solosky commented on a diff in the pull request: https://github.com/apache/incubator-rocketmq/pull/133#discussion_r132864458 --- Diff: common/src/main/java/org/apache/rocketmq/common/UtilAll.java --- @@ -184,15 +184,27 @@ public static String timeMillisToHumanString3(final long t) { cal.get(Calendar.SECOND)); } +/** + * Estimates the disk usage percentage by the specified path. + * + * @param path Path. + * @return Disk usage percentage by path. + */ public static double getDiskPartitionSpaceUsedPercent(final String path) { -if (null == path || path.isEmpty()) +if (null == path || path.isEmpty()) { --- End diff -- we should use StringUtils.isBlank(..) instead as a basic practice. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114190#comment-16114190 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user shroman commented on the issue: https://github.com/apache/incubator-rocketmq/pull/133 @zhouxinyu The problem is described in the JIRA issue -- "If disk usage estimates fail because the disk store path is not found (doesn't exist), message store falsely decides the disk is full and tries to delete." No failure, just a wasteful operation. I corrected the execution path, plus wrote the unit tests for it. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111995#comment-16111995 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user shroman commented on the issue: https://github.com/apache/incubator-rocketmq/pull/133 @vongosling @zhouxinyu @lizhanhui @vsair Anyone's willing to review the PR? > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100054#comment-16100054 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/133 [![Coverage Status](https://coveralls.io/builds/12539692/badge)](https://coveralls.io/builds/12539692) Coverage increased (+0.1%) to 38.817% when pulling **a4e9eb84f7ae4a68be59aae3e38b45c4bb595444 on shroman:ROCKETMQ-249** into **118bdec96005ce695295bcf0d9082e0230e69bf7 on apache:develop**. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100052#comment-16100052 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/133 [![Coverage Status](https://coveralls.io/builds/12539692/badge)](https://coveralls.io/builds/12539692) Coverage increased (+0.1%) to 38.817% when pulling **a4e9eb84f7ae4a68be59aae3e38b45c4bb595444 on shroman:ROCKETMQ-249** into **118bdec96005ce695295bcf0d9082e0230e69bf7 on apache:develop**. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098396#comment-16098396 ] ASF GitHub Bot commented on ROCKETMQ-249: - Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/133 [![Coverage Status](https://coveralls.io/builds/12521424/badge)](https://coveralls.io/builds/12521424) Coverage increased (+0.1%) to 38.836% when pulling **7b587210ff691fdaaecfe1a85f9fb3cc33b98ce2 on shroman:ROCKETMQ-249** into **118bdec96005ce695295bcf0d9082e0230e69bf7 on apache:develop**. > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ROCKETMQ-249) Attempt to clear disk even if disk store path is found
[ https://issues.apache.org/jira/browse/ROCKETMQ-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098320#comment-16098320 ] ASF GitHub Bot commented on ROCKETMQ-249: - GitHub user shroman opened a pull request: https://github.com/apache/incubator-rocketmq/pull/133 [ROCKETMQ-249] Do not attempt to clear disk even if disk store path i… …s found https://issues.apache.org/jira/browse/ROCKETMQ-249 You can merge this pull request into a Git repository by running: $ git pull https://github.com/shroman/incubator-rocketmq ROCKETMQ-249 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rocketmq/pull/133.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #133 commit 7b587210ff691fdaaecfe1a85f9fb3cc33b98ce2 Author: shromanDate: 2017-07-24T12:53:36Z [ROCKETMQ-249] Do not attempt to clear disk even if disk store path is found https://issues.apache.org/jira/browse/ROCKETMQ-249 > Attempt to clear disk even if disk store path is found > -- > > Key: ROCKETMQ-249 > URL: https://issues.apache.org/jira/browse/ROCKETMQ-249 > Project: Apache RocketMQ > Issue Type: Bug >Reporter: Roman Shtykh >Assignee: Roman Shtykh > > If disk usage estimates fail because the disk store path is not found > (doesn't exist), message store falsely decides the disk is full and tries to > delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)