[GitHub] incubator-rocketmq pull request #67: [ROCKETMQ-67] Consistent Hash allocate ...
Github user dhchao11 commented on a diff in the pull request:
https://github.com/apache/incubator-rocketmq/pull/67#discussion_r116650808
--- Diff:
common/src/main/java/org/apache/rocketmq/common/consistenthash/ConsistentHashRouter.java
---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.consistenthash;
+
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.SortedMap;
+import java.util.TreeMap;
+
+/**
+ * To hash Node objects to a hash ring with a certain amount of virtual
node.
+ * Method routeNode will return a Node instance which the object key
should be allocated to according to consistent hash algorithm
+ *
+ * @param
+ */
+public class ConsistentHashRouter {
+private final SortedMap ring = new TreeMap<>();
+private final HashFunction hashFunction;
+
+public ConsistentHashRouter(Collection pNodes, int vNodeCount) {
+this(pNodes,vNodeCount, new MD5Hash());
+}
+
+/**
+ *
+ * @param pNodes collections of physical nodes
+ * @param vNodeCount amounts of virtual nodes
+ * @param hashFunction hash Function to hash Node instances
+ */
+public ConsistentHashRouter(Collection pNodes, int vNodeCount,
HashFunction hashFunction) {
+if (hashFunction == null) {
+throw new NullPointerException("Hash Function is null");
+}
+this.hashFunction = hashFunction;
+if (pNodes != null) {
+for (T pNode : pNodes) {
+addNode(pNode, vNodeCount);
+}
+}
+}
+
+/**
+ * add physic node to the hash ring with some virtual nodes
+ * @param pNode physical node needs added to hash ring
+ * @param vNodeCount the number of virtual node of the physical node.
Value should be greater than or equals to 0
+ */
+public void addNode(T pNode, int vNodeCount) {
+if (vNodeCount < 0) throw new IllegalArgumentException("illegal
virtual node counts :" + vNodeCount);
+int existingReplicas = getExistingReplicas(pNode);
+for (int i = 0; i < vNodeCount; i++) {
+VirtualNode vNode = new VirtualNode<>(pNode, i +
existingReplicas);
+ring.put(hashFunction.hash(vNode.getKey()), vNode);
+}
+}
+
+/**
+ * remove the physical node from the hash ring
+ * @param pNode
+ */
+public void removeNode(T pNode) {
+Iterator it = ring.keySet().iterator();
+while (it.hasNext()) {
+Long key = it.next();
+VirtualNode virtualNode = ring.get(key);
+if (virtualNode.isVirtualNodeOf(pNode)) {
+it.remove();
+}
+}
+}
+
+/**
+ * with a specified key, route the nearest Node instance in the
current hash ring
+ * @param objectKey the object key to find a nearest Node
+ * @return
+ */
+public T routeNode(String objectKey) {
+if (ring.isEmpty()) {
+return null;
+}
+Long hashVal = hashFunction.hash(objectKey);
+SortedMap tailMap = ring.tailMap(hashVal);
+Long nodeHashVal = !tailMap.isEmpty() ? tailMap.firstKey() :
ring.firstKey();
+return ring.get(nodeHashVal).getPhysicalNode();
+}
+
+
+public int getExistingReplicas(T pNode) {
+int replicas = 0;
+for (VirtualNode vNode : ring.values()) {
+if (vNode.isVirtualNodeOf(pNode)) {
+replicas++;
+}
+}
+return replicas;
+}
+
+
+//default hash function
+private static class MD5Hash implements HashFunction {
+
[GitHub] incubator-rocketmq issue #60: [ROCKETMQ-96]Rename some temp variable and fie...
Github user Jaskey commented on the issue: https://github.com/apache/incubator-rocketmq/pull/60 @zhouxinyu @vongosling Any comments on this pr and any ideas about `hasTempMessage()`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rocketmq issue #101: [ROCKETMQ-194] log appender support
Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/101 [](https://coveralls.io/builds/11538511) Coverage increased (+0.3%) to 31.268% when pulling **fde2f7803500f4b4154c01c32dbfc5230e208805 on lindzh:log_appender** into **fa85abcdf6ee46e56403ddf86828a38683103fd0 on apache:master**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rocketmq issue #101: [ROCKETMQ-194] log appender support
Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/101 [](https://coveralls.io/builds/11538511) Coverage increased (+0.3%) to 31.268% when pulling **fde2f7803500f4b4154c01c32dbfc5230e208805 on lindzh:log_appender** into **fa85abcdf6ee46e56403ddf86828a38683103fd0 on apache:master**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rocketmq issue #101: [ROCKETMQ-194] log appender support
Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/101 [](https://coveralls.io/builds/11538511) Coverage increased (+0.3%) to 31.268% when pulling **fde2f7803500f4b4154c01c32dbfc5230e208805 on lindzh:log_appender** into **fa85abcdf6ee46e56403ddf86828a38683103fd0 on apache:master**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-rocketmq pull request #101: [ROCKETMQ-194] log appender support
Github user lindzh commented on a diff in the pull request:
https://github.com/apache/incubator-rocketmq/pull/101#discussion_r116641839
--- Diff:
logappender/src/main/java/org/apache/rocketmq/logappender/common/ProducerInstance.java
---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.logappender.common;
+
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.client.producer.DefaultMQProducer;
+import org.apache.rocketmq.client.producer.MQProducer;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Common Producer component
+ */
+public class ProducerInstance {
+
+public static final String APPENDER_TYPE = "APPENDER_TYPE";
+
+public static final String LOG4J_APPENDER = "LOG4J_APPENDER";
+
+public static final String LOG4J2_APPENDER = "LOG4J2_APPENDER";
+
+public static final String LOGBACK_APPENDER = "LOGBACK_APPENDER";
+
+public static final String DEFAULT_GROUP = "rocketmq_appender";
+
+private static ConcurrentHashMap producerMap = new
ConcurrentHashMap();
+
+private static String genKey(String nameServerAddress, String group) {
+return nameServerAddress + "_" + group;
+}
+
+
+public static MQProducer getInstance(String nameServerAddress, String
group) throws MQClientException {
+if (group == null) {
+group = DEFAULT_GROUP;
+}
+
+String genKey = genKey(nameServerAddress, group);
+MQProducer p = producerMap.get(genKey);
+if (p != null) {
+return p;
--- End diff --
As log initlizer is serializable,if it has ben started before,it must be
started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq pull request #101: [ROCKETMQ-194] log appender support
Github user lindzh commented on a diff in the pull request:
https://github.com/apache/incubator-rocketmq/pull/101#discussion_r116641020
--- Diff:
logappender/src/main/java/org/apache/rocketmq/logappender/common/ProducerInstance.java
---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.logappender.common;
+
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.client.producer.DefaultMQProducer;
+import org.apache.rocketmq.client.producer.MQProducer;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Common Producer component
+ */
+public class ProducerInstance {
+
+public static final String APPENDER_TYPE = "APPENDER_TYPE";
+
+public static final String LOG4J_APPENDER = "LOG4J_APPENDER";
+
+public static final String LOG4J2_APPENDER = "LOG4J2_APPENDER";
+
+public static final String LOGBACK_APPENDER = "LOGBACK_APPENDER";
+
+public static final String DEFAULT_GROUP = "rocketmq_appender";
+
+private static ConcurrentHashMap producerMap = new
ConcurrentHashMap();
+
+private static String genKey(String nameServerAddress, String group) {
+return nameServerAddress + "_" + group;
+}
+
+
+public static MQProducer getInstance(String nameServerAddress, String
group) throws MQClientException {
+if (group == null) {
+group = DEFAULT_GROUP;
+}
+
+String genKey = genKey(nameServerAddress, group);
+MQProducer p = producerMap.get(genKey);
+if (p != null) {
+return p;
+}
+
+DefaultMQProducer defaultMQProducer = new DefaultMQProducer(group);
+defaultMQProducer.setNamesrvAddr(nameServerAddress);
+MQProducer beforeProducer = null;
+//cas put producer
+synchronized (producerMap) {
--- End diff --
As log initlizer is serializable,there is no need to user synchronized
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq pull request #101: [ROCKETMQ-194] log appender support
Github user lindzh commented on a diff in the pull request:
https://github.com/apache/incubator-rocketmq/pull/101#discussion_r116638805
--- Diff:
logappender/src/main/java/org/apache/rocketmq/logappender/common/ProducerInstance.java
---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.logappender.common;
+
+import org.apache.rocketmq.client.exception.MQClientException;
+import org.apache.rocketmq.client.producer.DefaultMQProducer;
+import org.apache.rocketmq.client.producer.MQProducer;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Common Producer component
+ */
+public class ProducerInstance {
+
+public static final String APPENDER_TYPE = "APPENDER_TYPE";
+
+public static final String LOG4J_APPENDER = "LOG4J_APPENDER";
+
+public static final String LOG4J2_APPENDER = "LOG4J2_APPENDER";
+
+public static final String LOGBACK_APPENDER = "LOGBACK_APPENDER";
+
+public static final String DEFAULT_GROUP = "rocketmq_appender";
+
+private static ConcurrentHashMap producerMap = new
ConcurrentHashMap();
+
+private static String genKey(String nameServerAddress, String group) {
+return nameServerAddress + "_" + group;
+}
+
+
+public static MQProducer getInstance(String nameServerAddress, String
group) throws MQClientException {
+if (group == null) {
+group = DEFAULT_GROUP;
+}
+
+String genKey = genKey(nameServerAddress, group);
+MQProducer p = producerMap.get(genKey);
+if (p != null) {
+return p;
+}
+
+DefaultMQProducer defaultMQProducer = new DefaultMQProducer(group);
+defaultMQProducer.setNamesrvAddr(nameServerAddress);
+MQProducer beforeProducer = null;
+//cas put producer
+synchronized (producerMap) {
+beforeProducer = producerMap.putIfAbsent(genKey,
defaultMQProducer);
+}
+if (beforeProducer != null) {
+return beforeProducer;
+}
+defaultMQProducer.start();
+return defaultMQProducer;
+}
+
+
+public static void removeAndClose(String nameServerAddress, String
group) {
+if (group == null) {
+group = DEFAULT_GROUP;
+}
+String genKey = genKey(nameServerAddress, group);
+MQProducer producer = producerMap.get(genKey);
--- End diff --
Yes,makes it more simple.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rocketmq issue #60: [ROCKETMQ-96]Rename some temp variable and fie...
Github user shroman commented on the issue: https://github.com/apache/incubator-rocketmq/pull/60 @Jaskey No idea about `hasTempMessage()` -- no comments or unit tests... All the rest is ok to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
