[jira] [Commented] (ZOOKEEPER-3352) Use LevelDB For Backend
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815030#comment-16815030 ] maoling commented on ZOOKEEPER-3352: [~belugabehr] Plz take another look at the my referece provided above. - 1.There was a good evalutions from Xiang Li, Why the etcd-v3 choose the boltDB: https://github.com/etcd-io/etcd/issues/2646 - 2.Look at this: https://github.com/boltdb/bolt#comparison-with-other-databases *LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that they are libraries bundled into the application, however, their underlying structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes random writes by using a write ahead log and multi-tiered, sorted files called SSTables. Bolt uses a B+tree internally and only a single file. Both approaches have trade-offs.* *If you require a high random write throughput (>10,000 w/sec) or you need to use spinning disks then LevelDB could be a good choice. If your application is read-heavy or does a lot of range scans then Bolt could be a good choice.* *One other important consideration is that LevelDB does not have transactions. It supports batch writing of key/values pairs and it supports read snapshots but it will not give you the ability to do a compare-and-swap operation safely. Bolt supports fully serializable ACID transactions.* > Use LevelDB For Backend > --- > > Key: ZOOKEEPER-3352 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3352 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Critical > Fix For: 4.0.0 > > > Use LevelDB for managing data stored in ZK (transaction logs and snapshots). > https://stackoverflow.com/questions/6779669/does-leveldb-support-java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] maoling commented on issue #906: ZOOKEEPER-3360. Misprint in javadoc
maoling commented on issue #906: ZOOKEEPER-3360. Misprint in javadoc URL: https://github.com/apache/zookeeper/pull/906#issuecomment-481938481 The codes in the `recipes` is not a refined decoration one. In the further,we can remove lots of codes among them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (ZOOKEEPER-3360) Misprint in WriteLock javadoc
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ZOOKEEPER-3360:
--
Labels: newbie pull-request-available (was: newbie)
> Misprint in WriteLock javadoc
> -
>
> Key: ZOOKEEPER-3360
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3360
> Project: ZooKeeper
> Issue Type: Improvement
> Components: recipes
>Affects Versions: 3.5.5
> Environment: Any
>Reporter: Igor Rudenko
>Priority: Trivial
> Labels: newbie, pull-request-available
>
> {code:java}
> * @param acls the acls that you want to use for all the paths,
> {code}
> {code:java}
> public WriteLock(ZooKeeper zookeeper, String dir, List acl){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [zookeeper] hextriclosan opened a new pull request #906: ZOOKEEPER-3360. Misprint in javadoc
hextriclosan opened a new pull request #906: ZOOKEEPER-3360. Misprint in javadoc URL: https://github.com/apache/zookeeper/pull/906 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZOOKEEPER-3360) Misprint in WriteLock javadoc
Igor Rudenko created ZOOKEEPER-3360:
---
Summary: Misprint in WriteLock javadoc
Key: ZOOKEEPER-3360
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3360
Project: ZooKeeper
Issue Type: Improvement
Components: recipes
Affects Versions: 3.5.5
Environment: Any
Reporter: Igor Rudenko
{code:java}
* @param acls the acls that you want to use for all the paths,
{code}
{code:java}
public WriteLock(ZooKeeper zookeeper, String dir, List acl){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
ZooKeeper_branch34_openjdk7 - Build # 2271 - Failure
See https://builds.apache.org/job/ZooKeeper_branch34_openjdk7/2271/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 47.71 KB...] [junit] Running org.apache.zookeeper.test.SaslAuthFailNotifyTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.578 sec [junit] Running org.apache.zookeeper.test.SaslAuthFailTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.679 sec [junit] Running org.apache.zookeeper.test.SaslAuthMissingClientConfigTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.565 sec [junit] Running org.apache.zookeeper.test.SaslClientTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.074 sec [junit] Running org.apache.zookeeper.test.SessionInvalidationTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.546 sec [junit] Running org.apache.zookeeper.test.SessionTest [junit] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.6 sec [junit] Running org.apache.zookeeper.test.SessionTimeoutTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.748 sec [junit] Running org.apache.zookeeper.test.StandaloneTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.865 sec [junit] Running org.apache.zookeeper.test.StatTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.791 sec [junit] Running org.apache.zookeeper.test.StaticHostProviderTest [junit] Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.839 sec [junit] Running org.apache.zookeeper.test.SyncCallTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.681 sec [junit] Running org.apache.zookeeper.test.TruncateTest [junit] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.562 sec [junit] Running org.apache.zookeeper.test.UpgradeTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.203 sec [junit] Running org.apache.zookeeper.test.WatchedEventTest [junit] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.084 sec [junit] Running org.apache.zookeeper.test.WatcherFuncTest [junit] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.044 sec [junit] Running org.apache.zookeeper.test.WatcherTest [junit] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.793 sec [junit] Running org.apache.zookeeper.test.ZkDatabaseCorruptionTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.977 sec [junit] Running org.apache.zookeeper.test.ZooKeeperQuotaTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.635 sec [junit] Running org.apache.jute.BinaryInputArchiveTest [junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.073 sec fail.build.on.test.failure: BUILD FAILED /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build.xml:1425: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/ZooKeeper_branch34_openjdk7/build.xml:1428: Tests failed! Total time: 32 minutes 16 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Recording test results Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Email was triggered for: Failure - Any Sending email for trigger: Failure - Any Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ Setting OPENJDK_7_ON_UBUNTU_ONLY__HOME=/usr/lib/jvm/java-7-openjdk-amd64/ ### ## FAILED TESTS (if any) ## 1 tests failed. FAILED: org.apache.zookeeper.server.quorum.EphemeralNodeDeletionTest.testEphemeralNodeDeletion Error Message: After session close ephemeral node must be deleted expected null, but was:<4294967302,4294967302,1554923994335,1554923994335,0,0,0,145483729464393728,1,0,4294967302 > Stack Trace: junit.framework.AssertionFailedError: After session close ephemeral node must be deleted expected null, but was:<4294967302,4294967302,1554923994335,1554923994335,0,0,0,145483729464393728,1,0,4294967302 > at
[jira] [Updated] (ZOOKEEPER-3359) Batch commits in the CommitProcessor
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ZOOKEEPER-3359: -- Labels: pull-request-available (was: ) > Batch commits in the CommitProcessor > > > Key: ZOOKEEPER-3359 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3359 > Project: ZooKeeper > Issue Type: Improvement > Components: quorum >Affects Versions: 3.6.0 >Reporter: Brian Nixon >Priority: Minor > Labels: pull-request-available > > Draining a single commit every time the CommitProcessor switches to commit > mode can add to the backlog of committed messages. Instead, add controls to > batch and drain multiple commits and limit the number of reads being served. > Improves commit throughput and adds backpressure on reads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] enixon opened a new pull request #905: ZOOKEEPER-3359: Batch commits in the CommitProcessor
enixon opened a new pull request #905: ZOOKEEPER-3359: Batch commits in the CommitProcessor URL: https://github.com/apache/zookeeper/pull/905 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZOOKEEPER-3359) Batch commits in the CommitProcessor
Brian Nixon created ZOOKEEPER-3359: -- Summary: Batch commits in the CommitProcessor Key: ZOOKEEPER-3359 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3359 Project: ZooKeeper Issue Type: Improvement Components: quorum Affects Versions: 3.6.0 Reporter: Brian Nixon Draining a single commit every time the CommitProcessor switches to commit mode can add to the backlog of committed messages. Instead, add controls to batch and drain multiple commits and limit the number of reads being served. Improves commit throughput and adds backpressure on reads. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] eolivelli commented on a change in pull request #904: ZOOKEEPER-3037 - Add JVMPauseMonitor
eolivelli commented on a change in pull request #904: ZOOKEEPER-3037 - Add
JVMPauseMonitor
URL: https://github.com/apache/zookeeper/pull/904#discussion_r274080655
##
File path:
zookeeper-server/src/test/java/org/apache/zookeeper/server/util/JvmPauseMonitorTest.java
##
@@ -0,0 +1,133 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.util;
+
+import org.apache.zookeeper.server.ServerConfig;
+import org.apache.zookeeper.server.quorum.QuorumPeerConfig;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+public class JvmPauseMonitorTest {
+
+@Test
+public void testJvmPauseMonitorInit() {
+final Long sleepTime = 444L;
+final Long warnTH = L;
+final Long infoTH = 555L;
+
+ServerConfig serverConfig = new ServerConfig();
+QuorumPeerConfig quorumPeerConfig = new QuorumPeerConfig();
+
+Assert.assertEquals(false, serverConfig.isJvmPauseMonitorToRun());
+Assert.assertEquals(false, quorumPeerConfig.isJvmPauseMonitorToRun());
+
+Properties zkProp = new Properties();
+zkProp.setProperty("dataDir", new File("myDataDir").getAbsolutePath());
+zkProp.setProperty("jvm.pause.monitor", "true");
+zkProp.setProperty("jvm.pause.sleep.time.ms", sleepTime.toString());
+zkProp.setProperty("jvm.pause.warn-threshold.ms", warnTH.toString());
+zkProp.setProperty("jvm.pause.info-threshold.ms", infoTH.toString());
+try {
+quorumPeerConfig.parseProperties(zkProp);
+} catch (IOException | QuorumPeerConfig.ConfigException e) {
+Assert.fail("Exception while reading config for JvmPauseMonitor");
+}
+serverConfig.readFrom(quorumPeerConfig);
+
+Assert.assertEquals(true, serverConfig.isJvmPauseMonitorToRun());
+Assert.assertEquals(true, quorumPeerConfig.isJvmPauseMonitorToRun());
+
+JvmPauseMonitor pauseMonitor = new JvmPauseMonitor(serverConfig);
+Assert.assertFalse(pauseMonitor.isStarted());
+pauseMonitor.serviceStart();
+Assert.assertTrue(pauseMonitor.isStarted());
+
+Assert.assertEquals(sleepTime, Long.valueOf(pauseMonitor.sleepTimeMs));
+Assert.assertEquals(warnTH,
Long.valueOf(pauseMonitor.warnThresholdMs));
+Assert.assertEquals(infoTH,
Long.valueOf(pauseMonitor.infoThresholdMs));
+}
+
+@Test
+public void testJvmPauseMonitorExceedInfoThreshold() throws
InterruptedException {
+final Long sleepTime = 100L;
+final Long infoTH = -1L;
+
+ServerConfig serverConfig = new ServerConfig();
+QuorumPeerConfig quorumPeerConfig = new QuorumPeerConfig();
+
+Properties zkProp = new Properties();
+zkProp.setProperty("dataDir", new File("myDataDir").getAbsolutePath());
+zkProp.setProperty("jvm.pause.monitor", "true");
+zkProp.setProperty("jvm.pause.sleep.time.ms", sleepTime.toString());
+zkProp.setProperty("jvm.pause.info-threshold.ms", infoTH.toString());
+try {
+quorumPeerConfig.parseProperties(zkProp);
+} catch (IOException | QuorumPeerConfig.ConfigException e) {
+Assert.fail("Exception while reading config for JvmPauseMonitor");
+}
+serverConfig.readFrom(quorumPeerConfig);
+
+JvmPauseMonitor pauseMonitor = new JvmPauseMonitor(serverConfig);
+pauseMonitor.serviceStart();
+
+Assert.assertEquals(sleepTime, Long.valueOf(pauseMonitor.sleepTimeMs));
+Assert.assertEquals(infoTH,
Long.valueOf(pauseMonitor.infoThresholdMs));
+
+Thread.sleep(1000);
Review comment:
It is better to have a loop and wait at most X seconds for the condition to
be true.
This way we will reduce flakyness
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git
[GitHub] [zookeeper] eolivelli commented on a change in pull request #904: ZOOKEEPER-3037 - Add JVMPauseMonitor
eolivelli commented on a change in pull request #904: ZOOKEEPER-3037 - Add
JVMPauseMonitor
URL: https://github.com/apache/zookeeper/pull/904#discussion_r274080245
##
File path:
zookeeper-server/src/test/java/org/apache/zookeeper/server/util/JvmPauseMonitorTest.java
##
@@ -0,0 +1,133 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zookeeper.server.util;
+
+import org.apache.zookeeper.server.ServerConfig;
+import org.apache.zookeeper.server.quorum.QuorumPeerConfig;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Properties;
+
+public class JvmPauseMonitorTest {
+
+@Test
+public void testJvmPauseMonitorInit() {
+final Long sleepTime = 444L;
+final Long warnTH = L;
+final Long infoTH = 555L;
+
+ServerConfig serverConfig = new ServerConfig();
+QuorumPeerConfig quorumPeerConfig = new QuorumPeerConfig();
+
+Assert.assertEquals(false, serverConfig.isJvmPauseMonitorToRun());
+Assert.assertEquals(false, quorumPeerConfig.isJvmPauseMonitorToRun());
+
+Properties zkProp = new Properties();
+zkProp.setProperty("dataDir", new File("myDataDir").getAbsolutePath());
+zkProp.setProperty("jvm.pause.monitor", "true");
+zkProp.setProperty("jvm.pause.sleep.time.ms", sleepTime.toString());
+zkProp.setProperty("jvm.pause.warn-threshold.ms", warnTH.toString());
+zkProp.setProperty("jvm.pause.info-threshold.ms", infoTH.toString());
+try {
+quorumPeerConfig.parseProperties(zkProp);
+} catch (IOException | QuorumPeerConfig.ConfigException e) {
+Assert.fail("Exception while reading config for JvmPauseMonitor");
+}
+serverConfig.readFrom(quorumPeerConfig);
+
+Assert.assertEquals(true, serverConfig.isJvmPauseMonitorToRun());
+Assert.assertEquals(true, quorumPeerConfig.isJvmPauseMonitorToRun());
+
+JvmPauseMonitor pauseMonitor = new JvmPauseMonitor(serverConfig);
+Assert.assertFalse(pauseMonitor.isStarted());
+pauseMonitor.serviceStart();
+Assert.assertTrue(pauseMonitor.isStarted());
+
+Assert.assertEquals(sleepTime, Long.valueOf(pauseMonitor.sleepTimeMs));
+Assert.assertEquals(warnTH,
Long.valueOf(pauseMonitor.warnThresholdMs));
+Assert.assertEquals(infoTH,
Long.valueOf(pauseMonitor.infoThresholdMs));
+}
+
+@Test
+public void testJvmPauseMonitorExceedInfoThreshold() throws
InterruptedException {
+final Long sleepTime = 100L;
+final Long infoTH = -1L;
+
+ServerConfig serverConfig = new ServerConfig();
+QuorumPeerConfig quorumPeerConfig = new QuorumPeerConfig();
+
+Properties zkProp = new Properties();
+zkProp.setProperty("dataDir", new File("myDataDir").getAbsolutePath());
+zkProp.setProperty("jvm.pause.monitor", "true");
+zkProp.setProperty("jvm.pause.sleep.time.ms", sleepTime.toString());
+zkProp.setProperty("jvm.pause.info-threshold.ms", infoTH.toString());
+try {
+quorumPeerConfig.parseProperties(zkProp);
+} catch (IOException | QuorumPeerConfig.ConfigException e) {
+Assert.fail("Exception while reading config for JvmPauseMonitor");
+}
+serverConfig.readFrom(quorumPeerConfig);
+
+JvmPauseMonitor pauseMonitor = new JvmPauseMonitor(serverConfig);
+pauseMonitor.serviceStart();
Review comment:
Don't we have to stop it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (ZOOKEEPER-3037) Add JvmPauseMonitor to ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ZOOKEEPER-3037: -- Labels: pull-request-available (was: ) > Add JvmPauseMonitor to ZooKeeper > > > Key: ZOOKEEPER-3037 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3037 > Project: ZooKeeper > Issue Type: Improvement > Components: contrib >Affects Versions: 3.5.3, 3.4.12 >Reporter: Norbert Kalmar >Assignee: Norbert Kalmar >Priority: Minor > Labels: pull-request-available > > After a ZK crash, or client timeout sometimes it's hard to determine from the > logs what happened. Knowing if ZK was responsive at the time would help a > lot. For example, ZK might spend a lot of time waiting on GC (there is still > some misconception that ZK is a storage). > To help detect this, HADOOP already has a great tool called JVM Pause > Monitor. (As the name suggest, it can be also used for monitoring, but it > also helps post-mortem in a lot of cases). Basically it has a daemon that > sleeps for one second, and if the sleep time exceeds the 1s by more than the > threshold (1s: INFO, 10s: WARN by default - this can be configurable in our > case, see below), it will alert/make a log entry. It can also monitor the > time GC took. > The class implementing this is in HADOOP-common, but ZK should not depend on > this package. Since this is a straightforward implementation, and in the past > five years the few commits it had is nothing really serious, I think we could > just copy this class in ZooKeeper, and introduce it as a configurable > feature, by default it can be off. > The class: > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java > Task: > - Create a class in ZK (under zookeeper/server/util/) called JvmPauseMonitor. > - Make feature configurable, by default: OFF > - Make sleep time and threshold time configurable > - Update documentation > - Add [current size of the heap OR % of heap used] in the log entry whenever > sleep threshold had exceeded by a lot (10s) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] eolivelli commented on a change in pull request #854: ZOOKEEPER-3143 Pluggable metrics system for ZooKeeper - Data Collection on Server
eolivelli commented on a change in pull request #854: ZOOKEEPER-3143 Pluggable
metrics system for ZooKeeper - Data Collection on Server
URL: https://github.com/apache/zookeeper/pull/854#discussion_r274079313
##
File path:
zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java
##
@@ -392,8 +392,12 @@ public CommandResponse run(ZooKeeperServer zkServer,
Map kwargs)
response.put("observer_master_id",
((ObserverZooKeeperServer)zkServer).getObserver().getLearnerMasterId());
}
-response.putAll(ServerMetrics.getAllValues());
-
+ServerMetrics.getMetrics()
+.getMetricsProvider()
+.dump(
+(metric, value) -> {
Review comment:
btw As metrics are created at boot and they don't change I expect the order
not to change if you don't restart the server. At least for the
DefaultMetricsProvider.
@enixon
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zookeeper] enixon commented on a change in pull request #854: ZOOKEEPER-3143 Pluggable metrics system for ZooKeeper - Data Collection on Server
enixon commented on a change in pull request #854: ZOOKEEPER-3143 Pluggable
metrics system for ZooKeeper - Data Collection on Server
URL: https://github.com/apache/zookeeper/pull/854#discussion_r274077022
##
File path:
zookeeper-server/src/main/java/org/apache/zookeeper/server/admin/Commands.java
##
@@ -392,8 +392,12 @@ public CommandResponse run(ZooKeeperServer zkServer,
Map kwargs)
response.put("observer_master_id",
((ObserverZooKeeperServer)zkServer).getObserver().getLearnerMasterId());
}
-response.putAll(ServerMetrics.getAllValues());
-
+ServerMetrics.getMetrics()
+.getMetricsProvider()
+.dump(
+(metric, value) -> {
Review comment:
For any automated system, it probably doesn't matter. I'm just thinking that
a human operator would appreciate consistency between different iterations of
the command - particularly if the whole list gets long. :)
Not a blocker in any sense, just a convenience factor thought.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
ZooKeeper-trunk-windows-cmake - Build # 3294 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-windows-cmake/3294/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14.59 KB...] [move] Moving 1 file to f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\javacc\lib [javacc] Java Compiler Compiler Version 5.0 (Parser Generator) [javacc] (type "javacc" with no arguments for help) [javacc] Reading from file f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-jute\src\main\java\org\apache\jute\compiler\generated\rcc.jj . . . [javacc] File "TokenMgrError.java" does not exist. Will create one. [javacc] File "ParseException.java" does not exist. Will create one. [javacc] File "Token.java" does not exist. Will create one. [javacc] File "SimpleCharStream.java" does not exist. Will create one. [javacc] Parser generated successfully. jute: [javac] Compiling 39 source files to f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\build\classes [javac] warning: [options] bootstrap class path not set in conjunction with -source 1.8 [javac] 1 warning compile_jute_uptodate: compile_jute: [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-jute\target\main\java [mkdir] Created dir: f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-client\zookeeper-client-c\generated [java] ../../../src/main/resources/zookeeper.jute Parsed Successfully [java] ../../../zookeeper-jute/src/main/resources/zookeeper.jute Parsed Successfully [touch] Creating f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-jute\target\main\java\.generated BUILD SUCCESSFUL Total time: 24 seconds [ZooKeeper-trunk-windows-cmake] $ cmd /c call F:\jenkins\jenkins-slave\temp\jenkins5121498850785056103.bat f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake>cd zookeeper-client/zookeeper-client-c f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-client\zookeeper-client-c>cmake . -- Building for: Visual Studio 14 2015 -- The C compiler identification is unknown -- The CXX compiler identification is unknown CMake Error at CMakeLists.txt:19 (project): No CMAKE_C_COMPILER could be found. CMake Error at CMakeLists.txt:19 (project): No CMAKE_CXX_COMPILER could be found. -- Configuring incomplete, errors occurred! See also "F:/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-windows-cmake/zookeeper-client/zookeeper-client-c/CMakeFiles/CMakeOutput.log". See also "F:/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-windows-cmake/zookeeper-client/zookeeper-client-c/CMakeFiles/CMakeError.log". f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-client\zookeeper-client-c>cmake --build . Microsoft (R) Build Engine version 14.0.25420.1 Copyright (C) Microsoft Corporation. All rights reserved. MSBUILD : error MSB1009: Project file does not exist. Switch: ALL_BUILD.vcxproj f:\jenkins\jenkins-slave\workspace\ZooKeeper-trunk-windows-cmake\zookeeper-client\zookeeper-client-c>exit 1 Build step 'Execute Windows batch command' marked build as failure Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Commented] (ZOOKEEPER-3179) Add snapshot compression to reduce the disk IO
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814633#comment-16814633 ] Hudson commented on ZOOKEEPER-3179: --- SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #470 (See [https://builds.apache.org/job/ZooKeeper-trunk/470/]) ZOOKEEPER-3179: Add snapshot compression to reduce the disk IO (andor: rev 73cdd09051df33189351f7c42131724e3270eeac) * (edit) ivy.xml * (add) zookeeper-server/src/test/java/org/apache/zookeeper/server/persistence/SnapStreamTest.java * (add) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/SnapStream.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotFormatter.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileSnap.java * (edit) build.xml * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/Util.java * (edit) zookeeper-server/src/test/java/org/apache/zookeeper/server/ZooKeeperServerTest.java * (edit) zookeeper-server/pom.xml > Add snapshot compression to reduce the disk IO > -- > > Key: ZOOKEEPER-3179 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3179 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Fangmin Lv >Assignee: Yisong Yue >Priority: Major > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > When the snapshot becomes larger, the periodically snapshot after certain > number of txns will be more expensive. Which will in turn affect the maximum > throughput we can support within SLA, because of the disk contention between > snapshot and txn when they're on the same drive. > > With compression like zstd/snappy/gzip, the actual snapshot size could be > much smaller, the compress ratio depends on the actual data. It might make > the recovery time (loading from disk) faster in some cases, but will take > longer sometimes because of the extra time used to compress/decompress. > > Based on the production traffic, the performance various with different > compress method as well, that's why we provided different implementations, we > can select different compress method for different use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] anmolnar commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor
anmolnar commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor URL: https://github.com/apache/zookeeper/pull/904#issuecomment-481755763 @nkalmar Please add some description to the pull request or link the Jira ticket. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (ZOOKEEPER-3179) Add snapshot compression to reduce the disk IO
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814596#comment-16814596 ] Hudson commented on ZOOKEEPER-3179: --- SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #305 (See [https://builds.apache.org/job/Zookeeper-trunk-single-thread/305/]) ZOOKEEPER-3179: Add snapshot compression to reduce the disk IO (andor: rev 73cdd09051df33189351f7c42131724e3270eeac) * (add) zookeeper-server/src/test/java/org/apache/zookeeper/server/persistence/SnapStreamTest.java * (edit) zookeeper-server/src/test/java/org/apache/zookeeper/server/ZooKeeperServerTest.java * (edit) build.xml * (edit) ivy.xml * (edit) zookeeper-server/pom.xml * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotFormatter.java * (add) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/SnapStream.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/Util.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileSnap.java > Add snapshot compression to reduce the disk IO > -- > > Key: ZOOKEEPER-3179 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3179 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Fangmin Lv >Assignee: Yisong Yue >Priority: Major > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > When the snapshot becomes larger, the periodically snapshot after certain > number of txns will be more expensive. Which will in turn affect the maximum > throughput we can support within SLA, because of the disk contention between > snapshot and txn when they're on the same drive. > > With compression like zstd/snappy/gzip, the actual snapshot size could be > much smaller, the compress ratio depends on the actual data. It might make > the recovery time (loading from disk) faster in some cases, but will take > longer sometimes because of the extra time used to compress/decompress. > > Based on the production traffic, the performance various with different > compress method as well, that's why we provided different implementations, we > can select different compress method for different use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3179) Add snapshot compression to reduce the disk IO
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814549#comment-16814549 ] David Mollitor commented on ZOOKEEPER-3179: --- Also, requires update to docs. > Add snapshot compression to reduce the disk IO > -- > > Key: ZOOKEEPER-3179 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3179 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Fangmin Lv >Assignee: Yisong Yue >Priority: Major > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > When the snapshot becomes larger, the periodically snapshot after certain > number of txns will be more expensive. Which will in turn affect the maximum > throughput we can support within SLA, because of the disk contention between > snapshot and txn when they're on the same drive. > > With compression like zstd/snappy/gzip, the actual snapshot size could be > much smaller, the compress ratio depends on the actual data. It might make > the recovery time (loading from disk) faster in some cases, but will take > longer sometimes because of the extra time used to compress/decompress. > > Based on the production traffic, the performance various with different > compress method as well, that's why we provided different implementations, we > can select different compress method for different use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] eolivelli commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor
eolivelli commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor URL: https://github.com/apache/zookeeper/pull/904#issuecomment-481734690 anyway this is a great feature, thanks ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zookeeper] eolivelli commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor
eolivelli commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor URL: https://github.com/apache/zookeeper/pull/904#issuecomment-481734604 ok, I think that we should cite it in the header, like: https://github.com/apache/bookkeeper/blob/90c79444d6211cd66f4259c04c4e246097821b75/bookkeeper-server/src/test/java/org/apache/bookkeeper/sasl/MiniKdc.java#L67 even if it is an ASF project This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zookeeper] nkalmar commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor
nkalmar commented on issue #904: Zookeeper 3037 - Add JVMPauseMonitor URL: https://github.com/apache/zookeeper/pull/904#issuecomment-481723368 Yes, JVMPauseMonitor is taken out of Hadoop Common This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (ZOOKEEPER-3358) Make Snappy The Default Snapshot Compression Algorithm
David Mollitor created ZOOKEEPER-3358: - Summary: Make Snappy The Default Snapshot Compression Algorithm Key: ZOOKEEPER-3358 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3358 Project: ZooKeeper Issue Type: Improvement Components: server Affects Versions: 3.6.0 Reporter: David Mollitor Fix For: 3.6.0 Now that SnapShots are compressed, thanks to [ZOOKEEPER-3179], make the default algorithm Snappy compression. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3179) Add snapshot compression to reduce the disk IO
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814483#comment-16814483 ] David Mollitor commented on ZOOKEEPER-3179: --- Does this resolve ZOOKEEPER-1651 ? > Add snapshot compression to reduce the disk IO > -- > > Key: ZOOKEEPER-3179 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3179 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Fangmin Lv >Assignee: Yisong Yue >Priority: Major > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > When the snapshot becomes larger, the periodically snapshot after certain > number of txns will be more expensive. Which will in turn affect the maximum > throughput we can support within SLA, because of the disk contention between > snapshot and txn when they're on the same drive. > > With compression like zstd/snappy/gzip, the actual snapshot size could be > much smaller, the compress ratio depends on the actual data. It might make > the recovery time (loading from disk) faster in some cases, but will take > longer sometimes because of the extra time used to compress/decompress. > > Based on the production traffic, the performance various with different > compress method as well, that's why we provided different implementations, we > can select different compress method for different use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] BELUGABEHR commented on issue #903: ZOOKEEPER-3357: Remove Dead Link from ZooKeeper Programmer Guide
BELUGABEHR commented on issue #903: ZOOKEEPER-3357: Remove Dead Link from ZooKeeper Programmer Guide URL: https://github.com/apache/zookeeper/pull/903#issuecomment-481704508 @nkalmar Thanks. When I opened this PR originally, I was thinking there would be more than one dead link, making this PR more substantial, but in the end, I could only find the one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zookeeper] BELUGABEHR commented on a change in pull request #902: ZOOKEEPER-3355: Remove 'tbd' From Docs
BELUGABEHR commented on a change in pull request #902: ZOOKEEPER-3355: Remove 'tbd' From Docs URL: https://github.com/apache/zookeeper/pull/902#discussion_r273975803 ## File path: zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md ## @@ -1509,10 +1476,6 @@ ZooKeeper users fall into: Outside the formal documentation, there're several other sources of information for ZooKeeper developers. -* *ZooKeeper Whitepaper _[tbd: find url]_* : Review comment: I don't disagree, but I'm not sure the best way to include it. Can we simply include the PDF as a resource in the ZK documentation project itself so that there need be no concern about a dead link? I propose that this empty reference simply be deleted for now and that you open a new JIRA for figuring out the best way to add it in. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (ZOOKEEPER-3352) Use LevelDB For Backend
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814466#comment-16814466
]
David Mollitor commented on ZOOKEEPER-3352:
---
Ya, my drive here for using a third-party library on this is that ZK should not
be writing its own data store. ZK would benefit from a lot of the work that
these specialized projects are doing.
As a start, I was thinking that the ZK Transaction Log could be replaced with
Level/RocksDB.
* Compression: faster writes
* TTL for each record: TXN log entries expire automatically
Since TXN logs are not read that often (as I understand it), the 'read' buffer
for the DB would be disabled.
However, as an entire database system for ZK, I think these key/value stores
are probably a good fit for ZK since they keep everything sorted. For example,
for the following znodes, if the storage key is the path,
* /service/lock-001
* /service/lock-002
* /service/lock-003
To perform an 'ls' on the {{/service}} ZNode would only require a quick scan
> Use LevelDB For Backend
> ---
>
> Key: ZOOKEEPER-3352
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3352
> Project: ZooKeeper
> Issue Type: New Feature
> Components: server
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Critical
> Fix For: 4.0.0
>
>
> Use LevelDB for managing data stored in ZK (transaction logs and snapshots).
> https://stackoverflow.com/questions/6779669/does-leveldb-support-java
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3037) Add JvmPauseMonitor to ZooKeeper
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814416#comment-16814416 ] Norbert Kalmar commented on ZOOKEEPER-3037: --- Linking PT manually, looks like auto-linkage is not working? > Add JvmPauseMonitor to ZooKeeper > > > Key: ZOOKEEPER-3037 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3037 > Project: ZooKeeper > Issue Type: Improvement > Components: contrib >Affects Versions: 3.5.3, 3.4.12 >Reporter: Norbert Kalmar >Assignee: Norbert Kalmar >Priority: Minor > > After a ZK crash, or client timeout sometimes it's hard to determine from the > logs what happened. Knowing if ZK was responsive at the time would help a > lot. For example, ZK might spend a lot of time waiting on GC (there is still > some misconception that ZK is a storage). > To help detect this, HADOOP already has a great tool called JVM Pause > Monitor. (As the name suggest, it can be also used for monitoring, but it > also helps post-mortem in a lot of cases). Basically it has a daemon that > sleeps for one second, and if the sleep time exceeds the 1s by more than the > threshold (1s: INFO, 10s: WARN by default - this can be configurable in our > case, see below), it will alert/make a log entry. It can also monitor the > time GC took. > The class implementing this is in HADOOP-common, but ZK should not depend on > this package. Since this is a straightforward implementation, and in the past > five years the few commits it had is nothing really serious, I think we could > just copy this class in ZooKeeper, and introduce it as a configurable > feature, by default it can be off. > The class: > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java > Task: > - Create a class in ZK (under zookeeper/server/util/) called JvmPauseMonitor. > - Make feature configurable, by default: OFF > - Make sleep time and threshold time configurable > - Update documentation > - Add [current size of the heap OR % of heap used] in the log entry whenever > sleep threshold had exceeded by a lot (10s) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] nkalmar opened a new pull request #904: Zookeeper 3037 - Add JVMPauseMonitor
nkalmar opened a new pull request #904: Zookeeper 3037 - Add JVMPauseMonitor URL: https://github.com/apache/zookeeper/pull/904 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (ZOOKEEPER-3352) Use LevelDB For Backend
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maoling updated ZOOKEEPER-3352: --- Priority: Critical (was: Major) > Use LevelDB For Backend > --- > > Key: ZOOKEEPER-3352 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3352 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Critical > Fix For: 4.0.0 > > > Use LevelDB for managing data stored in ZK (transaction logs and snapshots). > https://stackoverflow.com/questions/6779669/does-leveldb-support-java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ZOOKEEPER-3352) Use LevelDB For Backend
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maoling updated ZOOKEEPER-3352: --- Issue Type: New Feature (was: Improvement) > Use LevelDB For Backend > --- > > Key: ZOOKEEPER-3352 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3352 > Project: ZooKeeper > Issue Type: New Feature > Components: server >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 4.0.0 > > > Use LevelDB for managing data stored in ZK (transaction logs and snapshots). > https://stackoverflow.com/questions/6779669/does-leveldb-support-java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3352) Use LevelDB For Backend
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814372#comment-16814372 ] maoling commented on ZOOKEEPER-3352: A good issue.That will be a huge work. 1. zk is read-heavy system,so LSM data structure may be not a good option. 2. For the reference the design of etcd-v3 which used boltDB(the https://github.com/boltdb/bolt) as the backend store engine. https://github.com/etcd-io/etcd/issues/2646 I also find this:mavibot, https://github.com/apache/directory-mavibot https://directory.apache.org/mavibot/user-guide.html 3.Before we choose the new backend store engine,we need to take the following into consideration: Does it support b-tree?, mvcc? Does it support tranctions? Is it light-weight, embedded? it has been endorsed by other famous system? > Use LevelDB For Backend > --- > > Key: ZOOKEEPER-3352 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3352 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 4.0.0 > > > Use LevelDB for managing data stored in ZK (transaction logs and snapshots). > https://stackoverflow.com/questions/6779669/does-leveldb-support-java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] asfgit closed pull request #690: ZOOKEEPER-3179: Add snapshot compression to reduce the disk IO
asfgit closed pull request #690: ZOOKEEPER-3179: Add snapshot compression to reduce the disk IO URL: https://github.com/apache/zookeeper/pull/690 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (ZOOKEEPER-3179) Add snapshot compression to reduce the disk IO
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andor Molnar resolved ZOOKEEPER-3179. - Resolution: Fixed Issue resolved by pull request 690 [https://github.com/apache/zookeeper/pull/690] > Add snapshot compression to reduce the disk IO > -- > > Key: ZOOKEEPER-3179 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3179 > Project: ZooKeeper > Issue Type: Improvement >Reporter: Fangmin Lv >Assignee: Yisong Yue >Priority: Major > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > When the snapshot becomes larger, the periodically snapshot after certain > number of txns will be more expensive. Which will in turn affect the maximum > throughput we can support within SLA, because of the disk contention between > snapshot and txn when they're on the same drive. > > With compression like zstd/snappy/gzip, the actual snapshot size could be > much smaller, the compress ratio depends on the actual data. It might make > the recovery time (loading from disk) faster in some cases, but will take > longer sometimes because of the extra time used to compress/decompress. > > Based on the production traffic, the performance various with different > compress method as well, that's why we provided different implementations, we > can select different compress method for different use cases. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [zookeeper] anmolnar commented on issue #690: ZOOKEEPER-3179: Add snapshot compression to reduce the disk IO
anmolnar commented on issue #690: ZOOKEEPER-3179: Add snapshot compression to reduce the disk IO URL: https://github.com/apache/zookeeper/pull/690#issuecomment-481649410 Landed to master branch. Thanks @yisong-yue ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
ZooKeeper 3.5.5-RC3
Hi all, I can't reply to previous mail because I'm not getting any emails from apache list. So I just checked the archive and registered with my other email to the list. To reply to some of the comments on RC3: I checked spotbugs and rat errors on the tag. I don't think it's a blocker either. rat gives 27 error of missing header, all should be in the exception list (which we do not have for maven yet - Enrico created the jira for it here: https://issues.apache.org/jira/browse/ZOOKEEPER-3351 ) So +1 from me on RC3:- unit tests OK- source build OK- start ZK on built source OK- run ZK from bin package OK- jar and license files looks OK (there are changes from 3.5.4) p.s.: I added the rat check to maven release procedure, depending on the outcome of ZOOKEEPER-3351 we can modify it later. Regards,Norbert
[GitHub] [zookeeper] nkalmar commented on a change in pull request #902: ZOOKEEPER-3355: Remove 'tbd' From Docs
nkalmar commented on a change in pull request #902: ZOOKEEPER-3355: Remove 'tbd' From Docs URL: https://github.com/apache/zookeeper/pull/902#discussion_r273883809 ## File path: zookeeper-docs/src/main/resources/markdown/zookeeperProgrammers.md ## @@ -1509,10 +1476,6 @@ ZooKeeper users fall into: Outside the formal documentation, there're several other sources of information for ZooKeeper developers. -* *ZooKeeper Whitepaper _[tbd: find url]_* : Review comment: We could add the white paper here? https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf Not sure if that's entirely legal though, but there's als the researchGate link: https://www.researchgate.net/publication/228366039_ZooKeeper_Wait-free_Coordination_for_Internet-scale_Systems Anyway, I'm also fine with just removing it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
