[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353568469
##
File path: fe/src/main/java/org/apache/doris/catalog/TableProperty.java
##
@@ -0,0 +1,94 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.clone.DynamicPartitionScheduler;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.common.util.TimeUtils;
+import org.json.JSONObject;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+public class TableProperty implements Writable {
+private Map properties = new HashMap<>();
+
+private static final String DYNAMIC_PARTITION_PROPERTY_PREFIX =
"dynamic_partition";
+private DynamicPartitionProperty dynamicPartitionProperty;
+
+public TableProperty() {}
+
+public TableProperty(Map properties) {
+this.properties = properties;
+}
+
+public Map getProperties() {
+return properties;
+}
+
+public DynamicPartitionProperty getDynamicPartitionProperty() {
+return dynamicPartitionProperty;
+}
+
+public void modifyTableProperties(Map modifyProperties) {
+properties.putAll(modifyProperties);
+this.dynamicPartitionProperty = buildDynamicProperty(properties);
+}
+
+private DynamicPartitionProperty buildDynamicProperty(Map
properties) {
+HashMap dynamicPartitionProperties = new HashMap<>();
+for (Map.Entry entry : properties.entrySet()) {
+if (entry.getKey().startsWith(DYNAMIC_PARTITION_PROPERTY_PREFIX)) {
+dynamicPartitionProperties.put(entry.getKey(),
entry.getValue());
+
DynamicPartitionScheduler.setLastUpdateTime(TimeUtils.getCurrentFormatTime());
+}
+}
+return new DynamicPartitionProperty(dynamicPartitionProperties);
+}
+
+public static TableProperty read(DataInput in) throws IOException {
+TableProperty info = new TableProperty();
+info.readFields(in);
+return info;
+}
+
+@Override
+public void write(DataOutput out) throws IOException {
+JSONObject jsonObject = new JSONObject();
Review comment:
There is a constructor of `JSONObject` you can use:
`JSONObject jsonObject = new JSONObject(properties);`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353568441
##
File path: fe/src/main/java/org/apache/doris/catalog/TableProperty.java
##
@@ -0,0 +1,94 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.clone.DynamicPartitionScheduler;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.common.util.TimeUtils;
+import org.json.JSONObject;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+public class TableProperty implements Writable {
+private Map properties = new HashMap<>();
+
+private static final String DYNAMIC_PARTITION_PROPERTY_PREFIX =
"dynamic_partition";
+private DynamicPartitionProperty dynamicPartitionProperty;
+
+public TableProperty() {}
+
+public TableProperty(Map properties) {
+this.properties = properties;
+}
+
+public Map getProperties() {
+return properties;
+}
+
+public DynamicPartitionProperty getDynamicPartitionProperty() {
+return dynamicPartitionProperty;
+}
+
+public void modifyTableProperties(Map modifyProperties) {
+properties.putAll(modifyProperties);
+this.dynamicPartitionProperty = buildDynamicProperty(properties);
+}
+
+private DynamicPartitionProperty buildDynamicProperty(Map
properties) {
+HashMap dynamicPartitionProperties = new HashMap<>();
+for (Map.Entry entry : properties.entrySet()) {
+if (entry.getKey().startsWith(DYNAMIC_PARTITION_PROPERTY_PREFIX)) {
+dynamicPartitionProperties.put(entry.getKey(),
entry.getValue());
+
DynamicPartitionScheduler.setLastUpdateTime(TimeUtils.getCurrentFormatTime());
Review comment:
why updating the `LastUpdateTime` here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353566293 ## File path: fe/src/main/java/org/apache/doris/catalog/Catalog.java ## @@ -3573,6 +3592,11 @@ private Table createOlapTable(Database db, CreateTableStmt stmt, boolean isResto PropertyAnalyzer.analyzeDataProperty(stmt.getProperties(), DataProperty.DEFAULT_HDD_DATA_PROPERTY); PropertyAnalyzer.analyzeReplicationNum(properties, FeConstants.default_replication_num); +Map dynamicPartitionProperties = DynamicPartitionUtil.analyzeDynamicPartition(db, olapTable, properties); Review comment: Here you call `analyzeDynamicPartition` before creating the partition. And inside the `analyzeDynamicPartition`, you will add this table to the `DynamicPartitionScheduler`. But if creating partition failed, no one remove this table out of `DynamicPartitionScheduler`, which causing memory leak. You should make sure that every is OK, then you can register this table to the `DynamicPartitionScheduler`. Or you need to find a way to clean it out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353567268
##
File path:
fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.clone;
+
+import org.apache.doris.analysis.AddPartitionClause;
+import org.apache.doris.analysis.DistributionDesc;
+import org.apache.doris.analysis.HashDistributionDesc;
+import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.analysis.PartitionValue;
+import org.apache.doris.analysis.SingleRangePartitionDesc;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.RangePartitionInfo;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableProperty;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.common.util.TimeUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class DynamicPartitionScheduler extends MasterDaemon {
+private static final Logger LOG =
LogManager.getLogger(DynamicPartitionScheduler.class);
+
+private static final String defaultValue = "N/A";
+private static String lastSchedulerTime = defaultValue;
+private static String lastUpdateTime = defaultValue;
+private static State dynamicPartitionState = State.NORMAL;
+private static String Msg = defaultValue;
+
+public enum State {
+NORMAL,
+ERROR
+}
+
+public static String getLastSchedulerTime() {
+return lastSchedulerTime;
+}
+
+public static void setLastSchedulerTime(String lastSchedulerTime) {
+DynamicPartitionScheduler.lastSchedulerTime = lastSchedulerTime;
+}
+
+public static String getLastUpdateTime() {
+return lastUpdateTime;
+}
+
+public static void setLastUpdateTime(String lastUpdateTime) {
+DynamicPartitionScheduler.lastUpdateTime = lastUpdateTime;
+}
+
+public static State getDynamicPartitionState() {
+return dynamicPartitionState;
+}
+
+public static void setDynamicPartitionState(State state) {
+dynamicPartitionState = state;
+}
+
+public static String getMsg() {
+return Msg;
+}
+
+public static void setMsg(String msg) {
+Msg = msg;
+}
+
+
+private static Set> dynamicPartitionTableInfo = new
HashSet<>();
+private boolean initialize;
+
+public DynamicPartitionScheduler(String name, long intervalMs) {
+super(name, intervalMs);
+this.initialize = false;
+}
+public synchronized static void registerDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.add(new Pair<>(dbId, tableId));
+}
+
+public synchronized static void removeDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.remove(new Pair<>(dbId, tableId));
+}
+
+
+
+private void dynamicAddPartition() {
+for (Pair tableInfo : dynamicPartitionTableInfo) {
+Long dbId = tableInfo.first;
+Long tableId = tableInfo.second;
+Database db = Catalog.getInstance().getDb(dbId);
Review comment:
hold database's lock before visiting members inside the database.
This is mentioned in my last review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353567175
##
File path:
fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.clone;
+
+import org.apache.doris.analysis.AddPartitionClause;
+import org.apache.doris.analysis.DistributionDesc;
+import org.apache.doris.analysis.HashDistributionDesc;
+import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.analysis.PartitionValue;
+import org.apache.doris.analysis.SingleRangePartitionDesc;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.RangePartitionInfo;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableProperty;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.common.util.TimeUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class DynamicPartitionScheduler extends MasterDaemon {
+private static final Logger LOG =
LogManager.getLogger(DynamicPartitionScheduler.class);
+
+private static final String defaultValue = "N/A";
+private static String lastSchedulerTime = defaultValue;
+private static String lastUpdateTime = defaultValue;
+private static State dynamicPartitionState = State.NORMAL;
+private static String Msg = defaultValue;
+
+public enum State {
+NORMAL,
+ERROR
+}
+
+public static String getLastSchedulerTime() {
+return lastSchedulerTime;
+}
+
+public static void setLastSchedulerTime(String lastSchedulerTime) {
+DynamicPartitionScheduler.lastSchedulerTime = lastSchedulerTime;
+}
+
+public static String getLastUpdateTime() {
+return lastUpdateTime;
+}
+
+public static void setLastUpdateTime(String lastUpdateTime) {
+DynamicPartitionScheduler.lastUpdateTime = lastUpdateTime;
+}
+
+public static State getDynamicPartitionState() {
+return dynamicPartitionState;
+}
+
+public static void setDynamicPartitionState(State state) {
+dynamicPartitionState = state;
+}
+
+public static String getMsg() {
+return Msg;
+}
+
+public static void setMsg(String msg) {
+Msg = msg;
+}
+
+
+private static Set> dynamicPartitionTableInfo = new
HashSet<>();
+private boolean initialize;
+
+public DynamicPartitionScheduler(String name, long intervalMs) {
+super(name, intervalMs);
+this.initialize = false;
+}
+public synchronized static void registerDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.add(new Pair<>(dbId, tableId));
+}
+
+public synchronized static void removeDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.remove(new Pair<>(dbId, tableId));
+}
+
+
+
+private void dynamicAddPartition() {
+for (Pair tableInfo : dynamicPartitionTableInfo) {
+Long dbId = tableInfo.first;
+Long tableId = tableInfo.second;
+Database db = Catalog.getInstance().getDb(dbId);
+if (db == null || db.getTable((tableId)) == null) {
+DynamicPartitionScheduler.removeDynamicPartitionTable(dbId,
tableId);
+continue;
+}
+
+// Determine the partition column type
+// if column type is Date, format partition name as MMdd
+// if column type is DateTime, format partition name as
MMddHHssmm
+
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353567126
##
File path:
fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.clone;
+
+import org.apache.doris.analysis.AddPartitionClause;
+import org.apache.doris.analysis.DistributionDesc;
+import org.apache.doris.analysis.HashDistributionDesc;
+import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.analysis.PartitionValue;
+import org.apache.doris.analysis.SingleRangePartitionDesc;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.RangePartitionInfo;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableProperty;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.common.util.TimeUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class DynamicPartitionScheduler extends MasterDaemon {
+private static final Logger LOG =
LogManager.getLogger(DynamicPartitionScheduler.class);
+
+private static final String defaultValue = "N/A";
+private static String lastSchedulerTime = defaultValue;
+private static String lastUpdateTime = defaultValue;
+private static State dynamicPartitionState = State.NORMAL;
+private static String Msg = defaultValue;
+
+public enum State {
+NORMAL,
+ERROR
+}
+
+public static String getLastSchedulerTime() {
+return lastSchedulerTime;
+}
+
+public static void setLastSchedulerTime(String lastSchedulerTime) {
+DynamicPartitionScheduler.lastSchedulerTime = lastSchedulerTime;
+}
+
+public static String getLastUpdateTime() {
+return lastUpdateTime;
+}
+
+public static void setLastUpdateTime(String lastUpdateTime) {
+DynamicPartitionScheduler.lastUpdateTime = lastUpdateTime;
+}
+
+public static State getDynamicPartitionState() {
+return dynamicPartitionState;
+}
+
+public static void setDynamicPartitionState(State state) {
+dynamicPartitionState = state;
+}
+
+public static String getMsg() {
+return Msg;
+}
+
+public static void setMsg(String msg) {
+Msg = msg;
+}
+
+
+private static Set> dynamicPartitionTableInfo = new
HashSet<>();
+private boolean initialize;
+
+public DynamicPartitionScheduler(String name, long intervalMs) {
+super(name, intervalMs);
+this.initialize = false;
+}
+public synchronized static void registerDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.add(new Pair<>(dbId, tableId));
+}
+
+public synchronized static void removeDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.remove(new Pair<>(dbId, tableId));
+}
+
+
+
+private void dynamicAddPartition() {
+for (Pair tableInfo : dynamicPartitionTableInfo) {
+Long dbId = tableInfo.first;
+Long tableId = tableInfo.second;
+Database db = Catalog.getInstance().getDb(dbId);
Review comment:
hold database's lock before visiting members inside the database.
This is mentioned in my last review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353564809
##
File path:
fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.clone;
+
+import org.apache.doris.analysis.AddPartitionClause;
+import org.apache.doris.analysis.DistributionDesc;
+import org.apache.doris.analysis.HashDistributionDesc;
+import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.analysis.PartitionValue;
+import org.apache.doris.analysis.SingleRangePartitionDesc;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.RangePartitionInfo;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableProperty;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.common.util.TimeUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class DynamicPartitionScheduler extends MasterDaemon {
+private static final Logger LOG =
LogManager.getLogger(DynamicPartitionScheduler.class);
+
+private static final String defaultValue = "N/A";
+private static String lastSchedulerTime = defaultValue;
+private static String lastUpdateTime = defaultValue;
+private static State dynamicPartitionState = State.NORMAL;
+private static String Msg = defaultValue;
+
+public enum State {
+NORMAL,
+ERROR
+}
+
+public static String getLastSchedulerTime() {
+return lastSchedulerTime;
+}
+
+public static void setLastSchedulerTime(String lastSchedulerTime) {
+DynamicPartitionScheduler.lastSchedulerTime = lastSchedulerTime;
+}
+
+public static String getLastUpdateTime() {
+return lastUpdateTime;
+}
+
+public static void setLastUpdateTime(String lastUpdateTime) {
+DynamicPartitionScheduler.lastUpdateTime = lastUpdateTime;
+}
+
+public static State getDynamicPartitionState() {
+return dynamicPartitionState;
+}
+
+public static void setDynamicPartitionState(State state) {
+dynamicPartitionState = state;
+}
+
+public static String getMsg() {
+return Msg;
+}
+
+public static void setMsg(String msg) {
+Msg = msg;
+}
+
+
+private static Set> dynamicPartitionTableInfo = new
HashSet<>();
+private boolean initialize;
+
+public DynamicPartitionScheduler(String name, long intervalMs) {
+super(name, intervalMs);
+this.initialize = false;
+}
+public synchronized static void registerDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.add(new Pair<>(dbId, tableId));
+}
+
+public synchronized static void removeDynamicPartitionTable(Long dbId,
Long tableId) {
+dynamicPartitionTableInfo.remove(new Pair<>(dbId, tableId));
+}
+
+
+
+private void dynamicAddPartition() {
+for (Pair tableInfo : dynamicPartitionTableInfo) {
+Long dbId = tableInfo.first;
+Long tableId = tableInfo.second;
+Database db = Catalog.getInstance().getDb(dbId);
+if (db == null || db.getTable((tableId)) == null) {
+DynamicPartitionScheduler.removeDynamicPartitionTable(dbId,
tableId);
+continue;
+}
+
+// Determine the partition column type
+// if column type is Date, format partition name as MMdd
+// if column type is DateTime, format partition name as
MMddHHssmm
+
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353568541
##
File path:
fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.clone;
+
+import org.apache.doris.analysis.AddPartitionClause;
+import org.apache.doris.analysis.DistributionDesc;
+import org.apache.doris.analysis.HashDistributionDesc;
+import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.analysis.PartitionValue;
+import org.apache.doris.analysis.SingleRangePartitionDesc;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.RangePartitionInfo;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableProperty;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.common.util.TimeUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class DynamicPartitionScheduler extends MasterDaemon {
+private static final Logger LOG =
LogManager.getLogger(DynamicPartitionScheduler.class);
+
+private static final String defaultValue = "N/A";
+private static String lastSchedulerTime = defaultValue;
+private static String lastUpdateTime = defaultValue;
+private static State dynamicPartitionState = State.NORMAL;
+private static String Msg = defaultValue;
Review comment:
```suggestion
private static String msg = defaultValue;
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353563769
##
File path:
fe/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
##
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.clone;
+
+import org.apache.doris.analysis.AddPartitionClause;
+import org.apache.doris.analysis.DistributionDesc;
+import org.apache.doris.analysis.HashDistributionDesc;
+import org.apache.doris.analysis.PartitionKeyDesc;
+import org.apache.doris.analysis.PartitionValue;
+import org.apache.doris.analysis.SingleRangePartitionDesc;
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.RangePartitionInfo;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableProperty;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.DynamicPartitionUtil;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.common.util.TimeUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Calendar;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class DynamicPartitionScheduler extends MasterDaemon {
+private static final Logger LOG =
LogManager.getLogger(DynamicPartitionScheduler.class);
+
+private static final String defaultValue = "N/A";
+private static String lastSchedulerTime = defaultValue;
+private static String lastUpdateTime = defaultValue;
+private static State dynamicPartitionState = State.NORMAL;
+private static String Msg = defaultValue;
+
+public enum State {
+NORMAL,
+ERROR
+}
+
+public static String getLastSchedulerTime() {
+return lastSchedulerTime;
+}
+
+public static void setLastSchedulerTime(String lastSchedulerTime) {
+DynamicPartitionScheduler.lastSchedulerTime = lastSchedulerTime;
+}
+
+public static String getLastUpdateTime() {
+return lastUpdateTime;
+}
+
+public static void setLastUpdateTime(String lastUpdateTime) {
+DynamicPartitionScheduler.lastUpdateTime = lastUpdateTime;
+}
+
+public static State getDynamicPartitionState() {
+return dynamicPartitionState;
+}
+
+public static void setDynamicPartitionState(State state) {
Review comment:
why using `static` method?. So does all other static methods.
`DynamicPartitionScheduler` is not some kind of "tool" class, like
'Utils.java'. So I think non-static method is better.
And I can't find other caller call this method outside the class, so maybe
`private` method is enough, or even remove this method, just set the state
using `dynamicPartitionState = xxx;` where you like
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353568436
##
File path: fe/src/main/java/org/apache/doris/catalog/TableProperty.java
##
@@ -0,0 +1,94 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog;
+
+import org.apache.doris.clone.DynamicPartitionScheduler;
+import org.apache.doris.common.io.Text;
+import org.apache.doris.common.io.Writable;
+import org.apache.doris.common.util.TimeUtils;
+import org.json.JSONObject;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+
+public class TableProperty implements Writable {
+private Map properties = new HashMap<>();
+
+private static final String DYNAMIC_PARTITION_PROPERTY_PREFIX =
"dynamic_partition";
+private DynamicPartitionProperty dynamicPartitionProperty;
+
+public TableProperty() {}
+
+public TableProperty(Map properties) {
+this.properties = properties;
+}
+
+public Map getProperties() {
+return properties;
+}
+
+public DynamicPartitionProperty getDynamicPartitionProperty() {
+return dynamicPartitionProperty;
+}
+
+public void modifyTableProperties(Map modifyProperties) {
+properties.putAll(modifyProperties);
+this.dynamicPartitionProperty = buildDynamicProperty(properties);
+}
+
+private DynamicPartitionProperty buildDynamicProperty(Map
properties) {
+HashMap dynamicPartitionProperties = new HashMap<>();
+for (Map.Entry entry : properties.entrySet()) {
+if (entry.getKey().startsWith(DYNAMIC_PARTITION_PROPERTY_PREFIX)) {
+dynamicPartitionProperties.put(entry.getKey(),
entry.getValue());
+
DynamicPartitionScheduler.setLastUpdateTime(TimeUtils.getCurrentFormatTime());
Review comment:
why updating the `LastUpdateTime` here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2367: [PROPOSAL]Support Dynamic Partition
morningman commented on a change in pull request #2367: [PROPOSAL]Support
Dynamic Partition
URL: https://github.com/apache/incubator-doris/pull/2367#discussion_r353573652
##
File path: fe/src/main/java/org/apache/doris/catalog/Catalog.java
##
@@ -5074,6 +5113,31 @@ public void replayRenameColumn(TableInfo tableInfo)
throws DdlException {
throw new DdlException("not implmented");
}
+public void modifyTableDynamicPartition(Database db, OlapTable table,
Map properties) throws DdlException {
Review comment:
The logic of modify dynamic partition is too simple.
There should be more check to validate the request.
For example, do we allow user to change the prefix of dynamic partition's
name? If yes, how to make sure that your partition checking logic in
`DynamicPartitionScheduler` works well after change?
As I know, You are judging whether to add the partition based on whether the
partition‘ name exists.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on issue #2142: 'nan' result didn't merge
yangzhg commented on issue #2142: 'nan' result didn't merge URL: https://github.com/apache/incubator-doris/issues/2142#issuecomment-561495344 这个问题的原因是在 be 和fe 的代码中我们对于数字和字符串的算数运算处理逻辑导致的，在字符串和数字做算数运算时 字符串会默认cast 成 double 类型。不同的数据库对于这个问题的处理也不同，比如postgres 不支持这种操作，mysql 支持，mysql 的逻辑是匹配字符串前缀 转换成数字，前缀没有数字则是0 比如 `1 + '1' = 2`, `1 + '1a' = 2` , `1 + 'a' = 1` , `1 + 'a1' = 2` , 我们目前的代码部分兼容了mysql 的操作，但是字符串转数字只支持完整的字符串格式和类似NAN/INF这种形式，其他情况会报错， 使整个表达式值变为 null，例如 `1 + '1' = 2`， `1 + '1a' = null` `1 + 'nan' = nan` 具体细节见代码 `be/src/util/string_parser.hpp`, 但是我们的代码似乎错用了string_parser.hpp， string_parser 的实际行为是和mysql 保持一致的， 只是在parse的状态上有 对于 完全是字符串的情况 会返回一个错误状态码，但是有默认值0返回，相反的对于NAN 和 INF 我们代码却没有对应的处理逻辑，应该返回NULL， 所以这个问题可以看成是doris 对于这种情况的一种错误处理 但是现在 由于doris 有一定的用户量，所以无论是改成 mysql 兼容还是 postgres 兼容 都会引起对现在的已有用户的不兼容现象，因此 这个问题可以暂时保持现状，等待以后在处理， 也可以修复这个错误的处理逻辑，因为对于使用了group by 的用户可能出现错误的结果 The reason for this problem is that in the code of be and fe, our arithmetic operation logic for numbers and strings is caused. When performing arithmetic operations on strings and numbers, strings are cast to double by default. Different databases deal with this problem differently. For example, postgres does not support this operation. MySQL supports it. The logic of mysql is to match the string prefix to a number. If the prefix is not a number, it is 0. For example, `1 + '1' = 2` , `1 + '1a' = 2`,` 1 + 'a' = 1`, `1 + 'a1' = 2`, our current code is partially compatible with mysql operations, but string to number support only complete String format and forms like NAN / INF, other cases will report errors, making the entire expression value null, such as `1 + '1' = 2`,` 1 + '1a' = null` `1 + 'nan' = nan` For details, see the code `be / src / util / string_parser.hpp`, but our code seems to use string_parser.hpp incorrectly. The actual behavior of string_parser is consistent with mysql, only in the parse state There is an error status code for a completely string case, but a default value of 0 is returned. In contrast, for NAN and INF, our code has no corresponding processing logic and should return NULL, so this problem can be seen as doris An error handling for this situation But now that doris has a certain number of users, whether it is mysql-compatible or postgres-compatible will cause incompatibility to existing users. Therefore, this problem can be temporarily maintained, waiting to be processed later, or you can fix this. Wrong processing logic, because users who use group by may have wrong results This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on issue #2145: when desc and asc，the result order of ‘’inf and nan‘’ is the same
yangzhg commented on issue #2145: when desc and asc，the result order of ‘’inf and nan‘’ is the same URL: https://github.com/apache/incubator-doris/issues/2145#issuecomment-561495497 the same with issue #2142 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 opened a new pull request #2374: Bug Fix: sql mode
xy720 opened a new pull request #2374: Bug Fix: sql mode URL: https://github.com/apache/incubator-doris/pull/2374 Issue #2365 This commit fixs the bug below, FE throws a unexpected exception when encounter a query like : Set sql_mode = '0,PIPES_AS_CONCAT'. and make some change to sql mode analyze process, now the analyze process is no longer put in SetVar.class, but in VariableMgr.class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg opened a new pull request #2362: [FIX] fix arithmetic operation between numeric and non-numeric
yangzhg opened a new pull request #2362: [FIX] fix arithmetic operation between numeric and non-numeric URL: https://github.com/apache/incubator-doris/pull/2362 fix arithmetic operation between numeric and non-numeric will cause unexpected value (#2142 #2145) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 closed pull request #2355: Bug fixs: sql mode
xy720 closed pull request #2355: Bug fixs: sql mode URL: https://github.com/apache/incubator-doris/pull/2355 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (0f00feb -> 9253627)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 0f00feb Optimize Doris On Elasticsearch performance (#2237) add 9253627 Fixed bdbje heartbeat timeout config format bug (#2369) No new revisions were added by this update. Summary of changes: fe/src/main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2369: Fixed bdbje heartbeat timeout config format bug
morningman merged pull request #2369: Fixed bdbje heartbeat timeout config format bug URL: https://github.com/apache/incubator-doris/pull/2369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] blackfox1983 opened a new issue #2373: null is not support in doris on es
blackfox1983 opened a new issue #2373: null is not support in doris on es URL: https://github.com/apache/incubator-doris/issues/2373 when use sql like 'select col from table where col is NULL' or 'select col from table where col is not NULL'. doris (on es) will 'match_all' on es. We should use 'exist query' syntax to push down to es to optimize the efficiency and effect of the whole query This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg removed a comment on issue #2142: 'nan' result didn't merge
yangzhg removed a comment on issue #2142: 'nan' result didn't merge URL: https://github.com/apache/incubator-doris/issues/2142#issuecomment-561471885 这个现象的原因是在 be 和fe 的代码中我们对于 数字和字符串的算数运算处理逻辑导致的，在字符串和数字做算数运算时 字符串会默认cast 成 double 类型。不同的数据库对于这个问题的处理也不同，比如postgres 不支持这种操作，mysql 支持，mysql 的逻辑是匹配字符串前缀 转换成数字，前缀没有数字则是0 比如 `1 + '1' = 2`, `1 + '1a' = 2` , `1 + 'a' = 1` , `1 + 'a1' = 2` , 我们部分兼容了mysql 的操作，但是字符串转数字 只支持完整的字符串格式， 和类似NAN 这种 其他情况会 报错， 使整个表达式值变为 null，例如 `1 + '1' = 2`， `1 + '1a' = null` `1 + 'nan' = nan` 具体细节见代码 `be/src/util/string_parser.hpp`, 但是现在 由于doris 有一定的用户量，所以无论是改成 mysql 兼容还是 postgres 兼容 都会引起对现在的已有用户的不兼容现象，因此 这个问题暂时保持现状，不做处理，等待以后在处理 The reason for this case is that in the code of be and fe, our logic of arithmetic operation for numbers and strings is caused. When performing arithmetic operations on strings and numbers, strings are cast to double by default. Different databases deal with this problem differently. For example, postgres does not support this operation. MySQL supports it. The logic of mysql is to match the string prefix to a number. If the prefix is not a number, it is 0. For example, 1 + '1' = 2, 1 + '1a' = 2, 1 + 'a' = 1, 1 + 'a1' = 2, we are partially compatible with mysql operations, but string to number only supports the full string format, and other similar to NAN and other The situation will report an error, making the entire expression value null, such as 1 + '1' = 2, 1 + '1a' = null 1 + 'nan' = nan. For details, see the code be / src / util / string_parser.hpp, But now that doris has a certain number of users, no matter whether it is mysql compatible or postgres compatible, it will cause incompatibility to existing users. Therefore, this issue will remain the status quo for the time being, do not deal with it, and wait for the future This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2237: Optimize Doris On Elasticsearch performance
morningman merged pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (f0c0a71 -> 0f00feb)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from f0c0a71 Add bdbje heartbeat timeout as a configuration of FE (#2366) add 0f00feb Optimize Doris On Elasticsearch performance (#2237) No new revisions were added by this update. Summary of changes: .gitignore | 2 +- be/src/exec/es/es_scan_reader.cpp | 2 +- be/src/exec/es/es_scroll_parser.cpp| 111 +--- be/src/exec/es/es_scroll_parser.h | 2 +- be/src/exec/es/es_scroll_query.cpp | 37 +++- be/src/exec/es/es_scroll_query.h | 2 +- be/src/exec/es_http_scan_node.cpp | 8 +- be/src/exec/es_http_scan_node.h| 1 + be/src/exec/es_http_scanner.cpp| 4 +- be/src/exec/es_http_scanner.h | 3 +- be/test/exec/es_scan_reader_test.cpp | 3 +- .../java/org/apache/doris/catalog/Catalog.java | 1 + .../java/org/apache/doris/catalog/EsTable.java | 197 + .../java/org/apache/doris/common/FeConstants.java | 2 +- .../org/apache/doris/common/FeMetaVersion.java | 6 + .../org/apache/doris/external/EsRestClient.java| 21 ++- .../org/apache/doris/external/EsStateStore.java| 145 ++- .../java/org/apache/doris/planner/EsScanNode.java | 4 + gensrc/thrift/PlanNodes.thrift | 18 ++ 19 files changed, 447 insertions(+), 122 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on issue #2142: 'nan' result didn't merge
yangzhg commented on issue #2142: 'nan' result didn't merge URL: https://github.com/apache/incubator-doris/issues/2142#issuecomment-561471885 这个现象的原因是在 be 和fe 的代码中我们对于 数字和字符串的算数运算处理逻辑导致的，在字符串和数字做算数运算时 字符串会默认cast 成 double 类型。不同的数据库对于这个问题的处理也不同，比如postgres 不支持这种操作，mysql 支持，mysql 的逻辑是匹配字符串前缀 转换成数字，前缀没有数字则是0 比如 `1 + '1' = 2`, `1 + '1a' = 2` , `1 + 'a' = 1` , `1 + 'a1' = 2` , 我们部分兼容了mysql 的操作，但是字符串转数字 只支持完整的字符串格式， 和类似NAN 这种 其他情况会 报错， 使整个表达式值变为 null，例如 `1 + '1' = 2`， `1 + '1a' = null` `1 + 'nan' = nan` 具体细节见代码 `be/src/util/string_parser.hpp`, 但是现在 由于doris 有一定的用户量，所以无论是改成 mysql 兼容还是 postgres 兼容 都会引起对现在的已有用户的不兼容现象，因此 这个问题暂时保持现状，不做处理，等待以后在处理 The reason for this case is that in the code of be and fe, our logic of arithmetic operation for numbers and strings is caused. When performing arithmetic operations on strings and numbers, strings are cast to double by default. Different databases deal with this problem differently. For example, postgres does not support this operation. MySQL supports it. The logic of mysql is to match the string prefix to a number. If the prefix is not a number, it is 0. For example, 1 + '1' = 2, 1 + '1a' = 2, 1 + 'a' = 1, 1 + 'a1' = 2, we are partially compatible with mysql operations, but string to number only supports the full string format, and other similar to NAN and other The situation will report an error, making the entire expression value null, such as 1 + '1' = 2, 1 + '1a' = null 1 + 'nan' = nan. For details, see the code be / src / util / string_parser.hpp, But now that doris has a certain number of users, no matter whether it is mysql compatible or postgres compatible, it will cause incompatibility to existing users. Therefore, this issue will remain the status quo for the time being, do not deal with it, and wait for the future This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2372: Modify the password and privilege checking logic
morningman opened a new pull request #2372: Modify the password and privilege checking logic URL: https://github.com/apache/incubator-doris/pull/2372 **Password checking logic** There are some problems with the current password and permission checking logic. For example: First, we create a user by: `create user cmy@"%" identified by "12345";` And then 'cmy' can login with password '12345' from any hosts. Second, we create another user by: `create user cmy@"192.168.%" identified by "abcde";` Because "192.168.%" has a higher priority in the permission table than "%". So when "cmy" try to login in by password "12345" from host "192.168.1.1", it should match the second permission entry, and will be rejected because of invalid password. But in current implementation, Doris will continue to check password on first entry, than let it pass. So we should change it. **Permission checking logic** After a user login, it should has a unique identity which is got from permission table. For example, when "cmy" from host "192.168.1.1" login, it's identity should be `cmy@"192.168.%"`. And Doris should use this identity to check other permission, not by using the user's real identity, which is `cmy@"192.168.1.1"`. **Black list** Functionally speaking, Doris only support adding WHITE LIST, which is to allow user to login from those hosts in the white list. But is some cases, we do need a BLACK LIST function. Fortunately, by changing the logic described above, we can simulate the effect of the BLACK LIST. For example, First we add a user by: `create user cmy@'%' identified by '12345';` And now user 'cmy' can login from any hosts. and if we don't want 'cmy' to login from host A, we can add a new user by: `create user cmy@'A' identified by 'other_passwd';` Because "A" has a higher priority in the permission table than "%". If 'cmy' try to login from A using password '12345', it will be rejected. ISSUE #2205 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] blackfox1983 opened a new issue #2371: debug mode in doris on es
blackfox1983 opened a new issue #2371: debug mode in doris on es URL: https://github.com/apache/incubator-doris/issues/2371 **Is your feature request related to a problem? Please describe.** when use doris on es, often need to view the corresponding DSL on a server according to SQL。iI need to scan all be'logs. It's very inefficient **Describe the solution you'd like** 1、Simple mode. On Doris'Web site, create a tab named debug。input a sql，then output the dsl。it's very useful for trace 2、Detail mode. input a sql，output data in every stage，e.g. dsl/some dataset scrolled from ES This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] jmk1011 opened a new issue #2370: some ut will occasionally fail
jmk1011 opened a new issue #2370: some ut will occasionally fail URL: https://github.com/apache/incubator-doris/issues/2370 some ut will occasionally fail ie. be: be/test/util/system_metrics_test.cpp:SystemMetricsTest.normal be/test/http/http_client_test.cpp fe: TableQueryPlanActionTest.testQueryPlanAction This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] yangzhg closed pull request #2362: [FIX] fix arithmetic operation between numeric and non-numeric
yangzhg closed pull request #2362: [FIX] fix arithmetic operation between numeric and non-numeric URL: https://github.com/apache/incubator-doris/pull/2362 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2369: Fixed bdbje heartbeat timeout config format bug
morningman opened a new pull request #2369: Fixed bdbje heartbeat timeout config format bug URL: https://github.com/apache/incubator-doris/pull/2369 The heartbeat config format should be like "30 s", not "30" This CL is related to commit 261072ecdda7e8eb3ce685c557c6dab15488d1f3 ISSUE #2357 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] blackfox1983 opened a new issue #2368: bug in sql:sleep
blackfox1983 opened a new issue #2368: bug in sql:sleep URL: https://github.com/apache/incubator-doris/issues/2368 select sleep(10), colfrom table limit 1; return just in ms... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2266: Modify the password and privilege checking logic
morningman commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353522955
##
File path: fe/src/main/java/org/apache/doris/mysql/privilege/UserPrivTable.java
##
@@ -141,37 +146,37 @@ public boolean checkPlainPassword(String remoteUser,
String remoteHost, String r
}
if (MysqlPassword.checkPlainPass(globalPrivEntry.getPassword(),
remotePasswd)) {
+if (currentUser != null) {
+if (entry.isSetByDomainResolver()) {
+currentUser.add(globalPrivEntry.getDomainUserIdent());
+} else {
+currentUser.add(entry.getUserIdent());
+}
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2266: Modify the password and privilege checking logic
morningman commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353519796
##
File path: fe/src/main/java/org/apache/doris/planner/SchemaScanNode.java
##
@@ -98,16 +99,17 @@ protected void toThrift(TPlanNode msg) {
if (schemaWild != null) {
msg.schema_scan_node.setWild(schemaWild);
}
-if (user != null) {
-msg.schema_scan_node.setUser(user);
-}
+
ConnectContext ctx = ConnectContext.get();
if (ctx != null) {
msg.schema_scan_node.setThread_id(ConnectContext.get().getConnectionId());
}
msg.schema_scan_node.setIp(frontendIP);
msg.schema_scan_node.setPort(frontendPort);
-msg.schema_scan_node.setUser_ip(userIp);
+
+UserIdentity currentUser =
ConnectContext.get().getCurrentUserIdentity();
+msg.schema_scan_node.setUser(currentUser.getQualifiedUser());
+msg.schema_scan_node.setIp(currentUser.getHost());
Review comment:
I will refactor this section. After that, the `user` and `user_ip` are
deprecated, and be replaced by a new member `current_user_ident`, which is type
of `TUserIdentity`.
Add a new type `TUserIdentity` because a user identity is not only composed
by `user` and
`ip`, but also has a property `is_domain`, to support BNS of DNS.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2266: Modify the password and privilege checking logic
morningman commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353519895
##
File path: fe/src/main/java/org/apache/doris/planner/SchemaScanNode.java
##
@@ -98,16 +99,17 @@ protected void toThrift(TPlanNode msg) {
if (schemaWild != null) {
msg.schema_scan_node.setWild(schemaWild);
}
-if (user != null) {
-msg.schema_scan_node.setUser(user);
-}
+
ConnectContext ctx = ConnectContext.get();
if (ctx != null) {
msg.schema_scan_node.setThread_id(ConnectContext.get().getConnectionId());
}
msg.schema_scan_node.setIp(frontendIP);
msg.schema_scan_node.setPort(frontendPort);
-msg.schema_scan_node.setUser_ip(userIp);
+
+UserIdentity currentUser =
ConnectContext.get().getCurrentUserIdentity();
+msg.schema_scan_node.setUser(currentUser.getQualifiedUser());
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed pull request #2266: Modify the password and privilege checking logic
morningman closed pull request #2266: Modify the password and privilege checking logic URL: https://github.com/apache/incubator-doris/pull/2266 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2266: Modify the password and privilege checking logic
morningman commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353507776
##
File path: fe/src/main/java/org/apache/doris/http/action/WebBaseAction.java
##
@@ -171,17 +175,21 @@ private boolean checkCookie(BaseRequest request,
BaseResponse response) {
String sessionId = request.getCookieValue(PALO_SESSION_ID);
HttpAuthManager authMgr = HttpAuthManager.getInstance();
if (!Strings.isNullOrEmpty(sessionId)) {
-String username = authMgr.getUsername(sessionId);
-if
(Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(request.getHostString(),
username,
+SessionValue sessionValue = authMgr.getSessionValue(sessionId);
+if (sessionValue == null) {
+return false;
+}
+if
(Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(sessionValue.currentUser,
PrivPredicate.of(PrivBitSet.of(PaloPrivilege.ADMIN_PRIV,
PaloPrivilege.NODE_PRIV),
Operator.OR))) {
response.updateCookieAge(request, PALO_SESSION_ID,
PALO_SESSION_EXPIRED_TIME);
request.setAuthorized(true);
ConnectContext ctx = new ConnectContext(null);
-ctx.setQualifiedUser(username);
+
ctx.setQualifiedUser(sessionValue.currentUser.getQualifiedUser());
ctx.setRemoteIP(request.getHostString());
+ctx.setCurrentUserIdentity(sessionValue.currentUser);
Review comment:
Not exactly, `currentUserIdentity` is the user identity got from priv table,
it is a "identity",
but `qualifiedUser` is just a user name. Currently, `qualifiedUser` is just
for some "show" information.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen closed issue #2062: Segement V2: SegmentIterator support bitmap index
kangkaisen closed issue #2062: Segement V2: SegmentIterator support bitmap index URL: https://github.com/apache/incubator-doris/issues/2062 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen closed issue #2333: Mac couldn't compile and run FE UT
kangkaisen closed issue #2333: Mac couldn't compile and run FE UT URL: https://github.com/apache/incubator-doris/issues/2333 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen closed issue #2061: Segement V2: Add bitmap index reader
kangkaisen closed issue #2061: Segement V2: Add bitmap index reader URL: https://github.com/apache/incubator-doris/issues/2061 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen closed issue #2336: Make doris more friendly for mac developers
kangkaisen closed issue #2336: Make doris more friendly for mac developers URL: https://github.com/apache/incubator-doris/issues/2336 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2366: Add bdbje heartbeat timeout as a configuration of FE
morningman merged pull request #2366: Add bdbje heartbeat timeout as a configuration of FE URL: https://github.com/apache/incubator-doris/pull/2366 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated: Add bdbje heartbeat timeout as a configuration of FE (#2366)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
The following commit(s) were added to refs/heads/master by this push:
new f0c0a71 Add bdbje heartbeat timeout as a configuration of FE (#2366)
f0c0a71 is described below
commit f0c0a715d1c5e63ec984e5159da9c0890da07ece
Author: Mingyu Chen
AuthorDate: Wed Dec 4 08:56:37 2019 +0800
Add bdbje heartbeat timeout as a configuration of FE (#2366)
The timeline for this question is as follows:
1. For some reason, the master have lost contact with the other two
followers.
Judging from the logs of the master, for almost 40 seconds, the master did
not print any logs.
It is suspected that it is stuck due to full gc or other reasons, causing
the
other two followers to think that the master has been disconnected.
2. After the other two followers re-elected, they continued to provide
services.
3. The master node is manually restarted afterwards. When restarting it for
the first time,
it needs to rollback some committed logs, so it needs to be closed and
restarted again.
After restarting again, it returns to normal.
The main reason is that the master got stuck for 40 seconds for some reason.
This issue requires further observation.
At the same time, in order to alleviate this problem, we decided to set
bdbje's heartbeat timeout
as a configurable value. The default is 30 seconds. Can be configured to 1
minute,
try to avoid this problem first.
---
fe/src/main/java/org/apache/doris/common/Config.java | 8
.../main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java | 2 ++
2 files changed, 10 insertions(+)
diff --git a/fe/src/main/java/org/apache/doris/common/Config.java
b/fe/src/main/java/org/apache/doris/common/Config.java
index b821886..6cb3c0f 100644
--- a/fe/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/src/main/java/org/apache/doris/common/Config.java
@@ -177,6 +177,14 @@ public class Config extends ConfigBase {
@ConfField public static String replica_ack_policy = "SIMPLE_MAJORITY"; //
ALL, NONE, SIMPLE_MAJORITY
/*
+ * The heartbeat timeout of bdbje between master and follower.
+ * the default is 30 seconds, which is same as default value in bdbje.
+ * If the network is experiencing transient problems, of some unexpected
long java GC annoying you,
+ * you can try to increase this value to decrease the chances of false
timeouts
+ */
+@ConfField public static int bdbje_heartbeat_timeout_second = 30;
+
+/*
* the max txn number which bdbje can rollback when trying to rejoin the
group
*/
@ConfField public static int txn_rollback_limit = 100;
diff --git
a/fe/src/main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java
b/fe/src/main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java
index f3ddc1d..7b50358 100644
--- a/fe/src/main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java
+++ b/fe/src/main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java
@@ -107,6 +107,8 @@ public class BDBEnvironment {
replicationConfig.setMaxClockDelta(Config.max_bdbje_clock_delta_ms,
TimeUnit.MILLISECONDS);
replicationConfig.setConfigParam(ReplicationConfig.TXN_ROLLBACK_LIMIT,
String.valueOf(Config.txn_rollback_limit));
+replicationConfig.setConfigParam(ReplicationConfig.REPLICA_TIMEOUT,
String.valueOf(Config.bdbje_heartbeat_timeout_second));
+replicationConfig.setConfigParam(ReplicationConfig.FEEDER_TIMEOUT,
String.valueOf(Config.bdbje_heartbeat_timeout_second));
if (isElectable) {
replicationConfig.setReplicaAckTimeout(2, TimeUnit.SECONDS);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (c8cff85 -> f828670)
This is an automated email from the ASF dual-hosted git repository. zhaoc pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from c8cff85 Fixed a bug that HttpServer in unit test does not start correctly. (#2361) add f828670 Add Bitmap index reader (#2319) No new revisions were added by this update. Summary of changes: be/src/exec/olap_scan_node.cpp | 4 + be/src/exec/olap_scan_node.h | 6 + be/src/exec/olap_scanner.cpp | 4 + be/src/olap/CMakeLists.txt | 1 + be/src/olap/column_block.h | 1 + be/src/olap/column_predicate.h | 10 + be/src/olap/comparison_predicate.cpp | 108 - be/src/olap/comparison_predicate.h | 1 + be/src/olap/field.h| 7 + be/src/olap/in_list_predicate.cpp | 49 be/src/olap/in_list_predicate.h| 3 + be/src/olap/null_predicate.cpp | 14 ++ be/src/olap/null_predicate.h | 6 + be/src/olap/olap_common.h | 4 + be/src/olap/rowset/beta_rowset_writer.cpp | 11 +- be/src/olap/rowset/beta_rowset_writer.h| 1 + .../olap/rowset/segment_v2/bitmap_index_reader.cpp | 52 .../olap/rowset/segment_v2/bitmap_index_reader.h | 102 +++- .../olap/rowset/segment_v2/bitmap_index_writer.cpp | 1 + be/src/olap/rowset/segment_v2/column_reader.cpp| 90 +++ be/src/olap/rowset/segment_v2/column_reader.h | 20 +- be/src/olap/rowset/segment_v2/index_page.cpp | 102 be/src/olap/rowset/segment_v2/index_page.h | 87 --- .../rowset/segment_v2/indexed_column_reader.cpp| 269 ++--- .../olap/rowset/segment_v2/indexed_column_reader.h | 88 +-- be/src/olap/rowset/segment_v2/page_pointer.h | 12 + be/src/olap/rowset/segment_v2/parsed_page.h| 64 + be/src/olap/rowset/segment_v2/row_ranges.h | 10 + be/src/olap/rowset/segment_v2/segment.cpp | 7 + be/src/olap/rowset/segment_v2/segment.h| 5 + be/src/olap/rowset/segment_v2/segment_iterator.cpp | 126 ++ be/src/olap/rowset/segment_v2/segment_iterator.h | 70 +- be/src/olap/rowset/segment_v2/segment_writer.cpp | 11 +- be/src/olap/rowset/segment_v2/segment_writer.h | 6 +- be/src/util/slice.h| 8 + be/test/olap/CMakeLists.txt| 1 + .../segment_v2/column_reader_writer_test.cpp | 14 +- .../segment_v2/index_column_reader_writer_test.cpp | 243 +++ be/test/olap/rowset/segment_v2/row_ranges_test.cpp | 29 +++ be/test/olap/rowset/segment_v2/segment_test.cpp| 203 ++-- gensrc/proto/segment_v2.proto | 1 - run-ut.sh | 1 + 42 files changed, 1561 insertions(+), 291 deletions(-) create mode 100644 be/src/olap/rowset/segment_v2/parsed_page.h create mode 100644 be/test/olap/rowset/segment_v2/index_column_reader_writer_test.cpp - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2266: Modify the password and privilege checking logic
imay commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353214383
##
File path: fe/src/main/java/org/apache/doris/mysql/privilege/PrivTable.java
##
@@ -52,28 +61,24 @@ public void addEntry(PrivEntry newEntry, boolean
errOnExist, boolean errOnNonExi
entries.add(newEntry);
Collections.sort(entries);
LOG.info("add priv entry: {}", newEntry);
+return newEntry;
} else {
if (errOnExist) {
throw new DdlException("User already exist");
} else {
-if (!checkOperationAllowed(existingEntry, newEntry, "ADD
ENTRY")) {
-return;
-} else {
-if (existingEntry.isSetByDomainResolver() &&
newEntry.isSetByDomainResolver()) {
-existingEntry.setPrivSet(newEntry.getPrivSet());
-LOG.debug("reset priv entry: {}", existingEntry);
-} else if (existingEntry.isSetByDomainResolver() &&
!newEntry.isSetByDomainResolver()
-|| !existingEntry.isSetByDomainResolver() &&
!newEntry.isSetByDomainResolver()) {
-mergePriv(existingEntry, newEntry);
-existingEntry.setSetByDomainResolver(false);
-LOG.info("merge priv entry: {}", existingEntry);
-}
-return;
+checkOperationAllowed(existingEntry, newEntry, "ADD ENTRY");
+if (existingEntry.isSetByDomainResolver() &&
newEntry.isSetByDomainResolver()) {
+existingEntry.setPrivSet(newEntry.getPrivSet());
+LOG.debug("reset priv entry: {}", existingEntry);
+} else if (existingEntry.isSetByDomainResolver() &&
!newEntry.isSetByDomainResolver()
+|| !existingEntry.isSetByDomainResolver() &&
!newEntry.isSetByDomainResolver()) {
Review comment:
is this condition equal with `!newEntry.isSetByDomainResolver()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2266: Modify the password and privilege checking logic
imay commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353147249
##
File path: fe/src/main/java/org/apache/doris/planner/SchemaScanNode.java
##
@@ -98,16 +99,17 @@ protected void toThrift(TPlanNode msg) {
if (schemaWild != null) {
msg.schema_scan_node.setWild(schemaWild);
}
-if (user != null) {
-msg.schema_scan_node.setUser(user);
-}
+
ConnectContext ctx = ConnectContext.get();
if (ctx != null) {
msg.schema_scan_node.setThread_id(ConnectContext.get().getConnectionId());
}
msg.schema_scan_node.setIp(frontendIP);
msg.schema_scan_node.setPort(frontendPort);
-msg.schema_scan_node.setUser_ip(userIp);
+
+UserIdentity currentUser =
ConnectContext.get().getCurrentUserIdentity();
+msg.schema_scan_node.setUser(currentUser.getQualifiedUser());
+msg.schema_scan_node.setIp(currentUser.getHost());
Review comment:
```suggestion
msg.schema_scan_node.setUser_ip(currentUser.getHost());
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2266: Modify the password and privilege checking logic
imay commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353141532
##
File path: fe/src/main/java/org/apache/doris/http/action/WebBaseAction.java
##
@@ -171,17 +175,21 @@ private boolean checkCookie(BaseRequest request,
BaseResponse response) {
String sessionId = request.getCookieValue(PALO_SESSION_ID);
HttpAuthManager authMgr = HttpAuthManager.getInstance();
if (!Strings.isNullOrEmpty(sessionId)) {
-String username = authMgr.getUsername(sessionId);
-if
(Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(request.getHostString(),
username,
+SessionValue sessionValue = authMgr.getSessionValue(sessionId);
+if (sessionValue == null) {
+return false;
+}
+if
(Catalog.getCurrentCatalog().getAuth().checkGlobalPriv(sessionValue.currentUser,
PrivPredicate.of(PrivBitSet.of(PaloPrivilege.ADMIN_PRIV,
PaloPrivilege.NODE_PRIV),
Operator.OR))) {
response.updateCookieAge(request, PALO_SESSION_ID,
PALO_SESSION_EXPIRED_TIME);
request.setAuthorized(true);
ConnectContext ctx = new ConnectContext(null);
-ctx.setQualifiedUser(username);
+
ctx.setQualifiedUser(sessionValue.currentUser.getQualifiedUser());
ctx.setRemoteIP(request.getHostString());
+ctx.setCurrentUserIdentity(sessionValue.currentUser);
Review comment:
looks like currentUserIdentity can cover qualifiedUser
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2266: Modify the password and privilege checking logic
imay commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353222580
##
File path: fe/src/main/java/org/apache/doris/mysql/privilege/UserPrivTable.java
##
@@ -141,37 +146,37 @@ public boolean checkPlainPassword(String remoteUser,
String remoteHost, String r
}
if (MysqlPassword.checkPlainPass(globalPrivEntry.getPassword(),
remotePasswd)) {
+if (currentUser != null) {
+if (entry.isSetByDomainResolver()) {
+currentUser.add(globalPrivEntry.getDomainUserIdent());
+} else {
+currentUser.add(entry.getUserIdent());
+}
Review comment:
Could you encapsulate this logic in getDomainUserIdent(), then it would be
not everywhere to judge this condition
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2266: Modify the password and privilege checking logic
imay commented on a change in pull request #2266: Modify the password and
privilege checking logic
URL: https://github.com/apache/incubator-doris/pull/2266#discussion_r353147850
##
File path: fe/src/main/java/org/apache/doris/planner/SchemaScanNode.java
##
@@ -98,16 +99,17 @@ protected void toThrift(TPlanNode msg) {
if (schemaWild != null) {
msg.schema_scan_node.setWild(schemaWild);
}
-if (user != null) {
-msg.schema_scan_node.setUser(user);
-}
+
ConnectContext ctx = ConnectContext.get();
if (ctx != null) {
msg.schema_scan_node.setThread_id(ConnectContext.get().getConnectionId());
}
msg.schema_scan_node.setIp(frontendIP);
msg.schema_scan_node.setPort(frontendPort);
-msg.schema_scan_node.setUser_ip(userIp);
+
+UserIdentity currentUser =
ConnectContext.get().getCurrentUserIdentity();
+msg.schema_scan_node.setUser(currentUser.getQualifiedUser());
Review comment:
If user and userIp is not used any more, better to remove them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo opened a new pull request #2367: [PROPOSAL]Support Dynamic Partition
WingsGo opened a new pull request #2367: [PROPOSAL]Support Dynamic Partition URL: https://github.com/apache/incubator-doris/pull/2367 For #2262 and #2317 1. Support dynamic partition when create an olap table. 2. Support modify dynamic partition properties. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] WingsGo closed pull request #2317: Support Dynamic Partition
WingsGo closed pull request #2317: Support Dynamic Partition URL: https://github.com/apache/incubator-doris/pull/2317 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 Pure DocValue optimization for doris-on-es https://github.com/apache/incubator-doris/issues/2048 Future todo: Today, for every tuple scan we check if pure_docvalue is enabled, this is not reasonable, should check pure_docvalue enabled for one whole scan outside, I will add this todo in future This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng closed pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng closed pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2366: Add bdbje heartbeat timeout as a configuration of FE
morningman opened a new pull request #2366: Add bdbje heartbeat timeout as a configuration of FE URL: https://github.com/apache/incubator-doris/pull/2366 The timeline for this question is as follows: 1. For some reason, the master have lost contact with the other two followers. Judging from the logs of the master, for almost 40 seconds, the master did not print any logs. It is suspected that it is stuck due to full gc or other reasons, causing the other two followers to think that the master has been disconnected. 2. After the other two followers re-elected, they continued to provide services. 3. The master node is manually restarted afterwards. When restarting it for the first time, it needs to rollback some committed logs, so it needs to be closed and restarted again. After restarting again, it returns to normal. The main reason is that the master got stuck for 40 seconds for some reason. This issue requires further observation. At the same time, in order to alleviate this problem, we decided to set bdbje's heartbeat timeout as a configurable value. The default is 30 seconds. Can be configured to 1 minute, try to avoid this problem first. ISSUE #2357 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (086bb82 -> c8cff85)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 086bb82 Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x (#2356) add c8cff85 Fixed a bug that HttpServer in unit test does not start correctly. (#2361) No new revisions were added by this update. Summary of changes: fe/src/main/java/org/apache/doris/http/HttpServer.java | 13 - .../test/java/org/apache/doris/http/DorisHttpTestCase.java | 4 +++- 2 files changed, 15 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly.
morningman merged pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly. URL: https://github.com/apache/incubator-doris/pull/2361 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #2357: Fe will notify new FE type transfer and quit unexpected
morningman edited a comment on issue #2357: Fe will notify new FE type transfer and quit unexpected URL: https://github.com/apache/incubator-doris/issues/2357#issuecomment-561145491 The timeline for this question is as follows: 1. For some reason, the master have lost contact with the other two followers . Judging from the logs of the master, for almost 40 seconds, the master did not print any logs. It is suspected that it is stuck due to full gc or other reasons, causing the other two followers to think that the master has been disconnected. 2. After the other two followers re-elected, they continued to provide services. 3. The master node is manually restarted afterwards. When restarting it for the first time, it needs to rollback some committed logs, so it needs to be closed and restarted again. After restarting again, it returns to normal. The main reason is that the master got stuck for 40 seconds for some reason. This issue requires further observation. At the same time, in order to alleviate this problem, we decided to set bdbje's heartbeat timeout as a configurable value. The default is 30 seconds. Can be configured to 1 minute, try to avoid this problem first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #2357: Fe will notify new FE type transfer and quit unexpected
morningman commented on issue #2357: Fe will notify new FE type transfer and quit unexpected URL: https://github.com/apache/incubator-doris/issues/2357#issuecomment-561145491 The timeline for this question is as follows: 1. For some reason, the master have lost contact with the other two followers . Judging from the logs of the master, for almost 40 seconds, the master did not print any logs. It is suspected that it is stuck due to full gc or other reasons, causing the other two followers to think that the master has been disconnected. 2. After the other two followers re-elected, they continued to provide services. 3. The master node is manually restarted afterwards. When restarting it for the first time, it needs to rollback some committed logs, so it needs to be closed and restarted again. After restarting again, it returns to normal. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r353144685
##
File path: be/src/olap/rowset/segment_v2/row_ranges.h
##
@@ -197,6 +199,14 @@ class RowRanges {
*result = std::move(tmp_range);
}
+static Roaring ranges_to_roaring(const RowRanges& ranges) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
kangkaisen commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r353144651
##
File path: be/src/olap/comparison_predicate.cpp
##
@@ -138,6 +140,90 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate,
<=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define BITMAP_COMPARE_EqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+RETURN_IF_ERROR(iterator->read_bitmap(seeked_ordinal, )); \
+} \
+} \
+} while (0)
+
+#define BITMAP_COMPARE_NotEqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (s.is_not_found()) { return Status::OK(); } \
+if (!s.ok()) { return s; } \
+if (!exact_match) { return Status::OK(); } \
+RETURN_IF_ERROR(iterator->read_bitmap(seeked_ordinal, )); \
+*bitmap -= roaring; \
+return Status::OK(); \
+} while (0)
+
+#define BITMAP_COMPARE_LessPredicate(s, exact_match, seeked_ordinal, iterator,
bitmap, roaring) do { \
+if (s.is_not_found()) { return Status::OK(); } \
+if (!s.ok()) { return s; } \
+RETURN_IF_ERROR(iterator->read_union_bitmap(0, seeked_ordinal, ));
\
+} while (0)
+
+#define BITMAP_COMPARE_LessEqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (s.is_not_found()) { return Status::OK(); } \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+seeked_ordinal++; \
+} \
+RETURN_IF_ERROR(iterator->read_union_bitmap(0, seeked_ordinal, ));
\
+} while (0)
+
+#define BITMAP_COMPARE_GreaterPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+seeked_ordinal++; \
+} \
+RETURN_IF_ERROR(iterator->read_union_bitmap(seeked_ordinal,
ordinal_limit, )); \
+} \
+} while (0)
+
+#define BITMAP_COMPARE_GreaterEqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+RETURN_IF_ERROR(iterator->read_union_bitmap(seeked_ordinal,
ordinal_limit, )); \
+} \
+} while (0)
+
+#define BITMAP_COMPARE(CLASS, s, exact_match, seeked_ordinal, iterator,
bitmap, roaring) \
+BITMAP_COMPARE_##CLASS(s, exact_match, seeked_ordinal, iterator,
bitmap, roaring)
+
+#define COMPARISON_PRED_BITMAP_EVALUATE(CLASS, OP) \
+template \
+Status CLASS::evaluate(const Schema& schema, const
vector& iterators, uint32_t num_rows, Roaring* bitmap)
const { \
+BitmapIndexIterator *iterator = iterators[_column_id]; \
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull request #2319: Add Bitmap index reader
kangpinghuang commented on a change in pull request #2319: Add Bitmap index
reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r353063001
##
File path: be/src/olap/comparison_predicate.cpp
##
@@ -138,6 +140,90 @@ COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(LessEqualPredicate,
<=)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterPredicate, >)
COMPARISON_PRED_COLUMN_BLOCK_EVALUATE(GreaterEqualPredicate, >=)
+#define BITMAP_COMPARE_EqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+RETURN_IF_ERROR(iterator->read_bitmap(seeked_ordinal, )); \
+} \
+} \
+} while (0)
+
+#define BITMAP_COMPARE_NotEqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (s.is_not_found()) { return Status::OK(); } \
+if (!s.ok()) { return s; } \
+if (!exact_match) { return Status::OK(); } \
+RETURN_IF_ERROR(iterator->read_bitmap(seeked_ordinal, )); \
+*bitmap -= roaring; \
+return Status::OK(); \
+} while (0)
+
+#define BITMAP_COMPARE_LessPredicate(s, exact_match, seeked_ordinal, iterator,
bitmap, roaring) do { \
+if (s.is_not_found()) { return Status::OK(); } \
+if (!s.ok()) { return s; } \
+RETURN_IF_ERROR(iterator->read_union_bitmap(0, seeked_ordinal, ));
\
+} while (0)
+
+#define BITMAP_COMPARE_LessEqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (s.is_not_found()) { return Status::OK(); } \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+seeked_ordinal++; \
+} \
+RETURN_IF_ERROR(iterator->read_union_bitmap(0, seeked_ordinal, ));
\
+} while (0)
+
+#define BITMAP_COMPARE_GreaterPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+if (exact_match) { \
+seeked_ordinal++; \
+} \
+RETURN_IF_ERROR(iterator->read_union_bitmap(seeked_ordinal,
ordinal_limit, )); \
+} \
+} while (0)
+
+#define BITMAP_COMPARE_GreaterEqualPredicate(s, exact_match, seeked_ordinal,
iterator, bitmap, roaring) do { \
+if (!s.is_not_found()) { \
+if (!s.ok()) { return s; } \
+RETURN_IF_ERROR(iterator->read_union_bitmap(seeked_ordinal,
ordinal_limit, )); \
+} \
+} while (0)
+
+#define BITMAP_COMPARE(CLASS, s, exact_match, seeked_ordinal, iterator,
bitmap, roaring) \
+BITMAP_COMPARE_##CLASS(s, exact_match, seeked_ordinal, iterator,
bitmap, roaring)
+
+#define COMPARISON_PRED_BITMAP_EVALUATE(CLASS, OP) \
+template \
+Status CLASS::evaluate(const Schema& schema, const
vector& iterators, uint32_t num_rows, Roaring* bitmap)
const { \
+BitmapIndexIterator *iterator = iterators[_column_id]; \
Review comment:
```suggestion
BitmapIndexIterator* iterator = iterators[_column_id]; \
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull request #2319: Add Bitmap index reader
kangpinghuang commented on a change in pull request #2319: Add Bitmap index
reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r353059091
##
File path: be/src/exec/olap_scan_node.h
##
@@ -273,6 +274,10 @@ class OlapScanNode : public ScanNode {
// page read from cache
// used by segment v2
RuntimeProfile::Counter* _cached_pages_num_counter = nullptr;
+
+RuntimeProfile::Counter* _bitmap_index_filter_counter = nullptr;
+RuntimeProfile::Counter* _bitmap_index_filter_timer = nullptr;
+RuntimeProfile::Counter* _predicate_count = nullptr;
Review comment:
what is this counter for?
add some comments?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull request #2319: Add Bitmap index reader
kangpinghuang commented on a change in pull request #2319: Add Bitmap index
reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r353073666
##
File path: be/src/olap/rowset/segment_v2/row_ranges.h
##
@@ -197,6 +199,14 @@ class RowRanges {
*result = std::move(tmp_range);
}
+static Roaring ranges_to_roaring(const RowRanges& ranges) {
Review comment:
add ut
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 Pure DocValue optimization for doris-on-es https://github.com/apache/incubator-doris/issues/2048 Future todo: Today, for every tuple scan we check if pure_docvalue is enabled, this is not reasonable, should check pure_docvalue enabled for one whole scan outside, I will add this todo in future This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 Pure DocValue optimization for doris-on-es https://github.com/apache/incubator-doris/issues/2048 Future todo: Today, for every tuple scan we check if pure_docvalue is enabled, this is not reasonable, should check pure_docvalue enabled for one whole scan outside, I will add this todo in future This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed pull request #2237: Optimize Doris On Elasticsearch performance
morningman closed pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng closed pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng closed pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly.
morningman closed pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly. URL: https://github.com/apache/incubator-doris/pull/2361 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng opened a new pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 Pure DocValue optimization for doris-on-es https://github.com/apache/incubator-doris/issues/2048 Future todo: Today, for every tuple scan we check if pure_docvalue is enabled, this is not reasonable, should check pure_docvalue enabled for one whole scan outside, I will add this todo in future This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] wuyunfeng closed pull request #2237: Optimize Doris On Elasticsearch performance
wuyunfeng closed pull request #2237: Optimize Doris On Elasticsearch performance URL: https://github.com/apache/incubator-doris/pull/2237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly.
morningman opened a new pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly. URL: https://github.com/apache/incubator-doris/pull/2361 Because the http client in unit test try to connect to the server when server is not ready yet. Add a boolean flag `isStarted` to indicate that the server thread is ready or not. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman closed pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly.
morningman closed pull request #2361: Fixed a bug that HttpServer in unit test does not start correctly. URL: https://github.com/apache/incubator-doris/pull/2361 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] xy720 opened a new issue #2365: Bug in setting sql mode
xy720 opened a new issue #2365: Bug in setting sql mode URL: https://github.com/apache/incubator-doris/issues/2365 A bug is found in setting mode: SET sql_mode='0,STRICT_TRANS_TABLES' is not supported. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #2294: Add convert tablet rowset type
chaoyli commented on a change in pull request #2294: Add convert tablet rowset
type
URL: https://github.com/apache/incubator-doris/pull/2294#discussion_r353052080
##
File path: fe/src/main/java/org/apache/doris/alter/RollupHandler.java
##
@@ -406,6 +413,48 @@ public void processDropRollup(DropRollupClause
alterClause, Database db, OlapTab
LOG.info("finished drop rollup index[{}] in table[{}]",
rollupIndexName, olapTable.getName());
}
+// sql: alter table base_table_name set property ("storage_format" = "v2");
Review comment:
This grammar is confusing. I think that grammar is used to changing all base
table to v2 format.
But is creating rollup actually.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2358: Remove VersionHash used to comparison in BE
morningman commented on a change in pull request #2358: Remove VersionHash used
to comparison in BE
URL: https://github.com/apache/incubator-doris/pull/2358#discussion_r353046045
##
File path: be/src/olap/schema_change.cpp
##
@@ -1932,10 +1932,8 @@ OLAPStatus
SchemaChangeHandler::_validate_alter_result(TabletSharedPtr new_table
new_tablet->max_continuous_version_from_begining(_continuous_version,
_continuous_version_hash);
LOG(INFO) << "find max continuous version of tablet=" <<
new_tablet->full_name()
<< ", start_version=" << max_continuous_version.first
- << ", end_version=" << max_continuous_version.second
- << ", version_hash=" << max_continuous_version_hash;
-if (max_continuous_version.second > request.alter_version
-|| (max_continuous_version.second == request.alter_version &&
max_continuous_version_hash == request.alter_version_hash)) {
+ << ", end_version=" << max_continuous_version.second;
+if (max_continuous_version.second > request.alter_version) {
Review comment:
```suggestion
if (max_continuous_version.second >= request.alter_version) {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
imay commented on a change in pull request #2364: add a control framework
between FE and BE through heartbeat #2247
URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353039573
##
File path: fe/src/main/java/org/apache/doris/qe/HeartbeatFlagHelper.java
##
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Maps;
+import org.apache.doris.common.AnalysisException;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+public class HeartbeatFlagHelper {
+
+/* Bits for different Heartbeat Flag modes, you can add custom flags here
*/
+public static final long FLAG_SET_DEFAULT_ROWSET_TYPE_TO_BETA = 1L;
Review comment:
should define this flag in thrift to avoid define it multiple times.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
imay commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247 URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353043690 ## File path: fe/src/main/java/org/apache/doris/qe/GlobalVariable.java ## @@ -56,6 +56,11 @@ @VariableMgr.VarAttr(name = "query_cache_size") private static long queryCacheSize = 1048576; +public static final String HEARTBEAT_FLAGS = "heartbeat_flags"; +// the flags passed to Backends +@VariableMgr.VarAttr(name = "heartbeat_flags") +public static long heartbeatFlags = 0; Review comment: I don't think it is a good idea to set this in one global variables. Maybe each single function has a corresponding variable. Because if there are some flags and you want to change one of them, then you should select all of them and change the result and set it back. This is error prone and not easy to use. And may be some flag in the future can't be changed through variables. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
imay commented on a change in pull request #2364: add a control framework
between FE and BE through heartbeat #2247
URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353046947
##
File path: be/src/agent/heartbeat_server.cpp
##
@@ -147,6 +147,10 @@ Status HeartbeatServer::_heartbeat(
_master_info->__set_http_port(master_info.http_port);
}
+if (master_info.__isset.heartbeat_flag) {
+_olap_engine->set_heartbeat_flag(master_info.heartbeat_flag);
Review comment:
I prefer that we store this flag in exec_env, class who has the requirement
access the flag it needs.
If changing in your way, there is no place to see what's the flags is. And
what's more, if someone needs this flags, it will be set here, which is high
coupling.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
imay commented on a change in pull request #2364: add a control framework
between FE and BE through heartbeat #2247
URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353044785
##
File path: fe/src/main/java/org/apache/doris/qe/HeartbeatFlagHelper.java
##
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Maps;
+import org.apache.doris.common.AnalysisException;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+public class HeartbeatFlagHelper {
Review comment:
May be what we need is a collector of flags. When some call getFlags, then
this class construct on e flags from many places, such as global variables, or
may be hard-coded.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
imay commented on a change in pull request #2364: add a control framework
between FE and BE through heartbeat #2247
URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353046941
##
File path: be/src/agent/heartbeat_server.cpp
##
@@ -147,6 +147,10 @@ Status HeartbeatServer::_heartbeat(
_master_info->__set_http_port(master_info.http_port);
}
+if (master_info.__isset.heartbeat_flag) {
+_olap_engine->set_heartbeat_flag(master_info.heartbeat_flag);
Review comment:
I prefer that we store this flag in exec_env, class who has the requirement
access the flag it needs.
If changing in your way, there is no place to see what's the flags is. And
what's more, if someone needs this flags, it will be set here, which is high
coupling.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
morningman commented on a change in pull request #2364: add a control framework
between FE and BE through heartbeat #2247
URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353042239
##
File path: fe/src/main/java/org/apache/doris/analysis/SetVar.java
##
@@ -128,6 +127,17 @@ else if (result instanceof IntLiteral) {
result = new
IntLiteral(SqlModeHelper.encode(sqlMode).toString(), Type.BIGINT);
}
}
+if (variable.equalsIgnoreCase(GlobalVariable.HEARTBEAT_FLAGS)) {
Review comment:
The way of analyzing variables has been changed.
You'd better wait this PR merged and change it to that way:
https://github.com/apache/incubator-doris/pull/2355
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
morningman commented on a change in pull request #2364: add a control framework
between FE and BE through heartbeat #2247
URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353040483
##
File path: be/src/olap/schema_change.cpp
##
@@ -1123,6 +1123,9 @@ bool SchemaChangeWithSorting::_internal_sorting(const
vector& row_blo
context.partition_id = new_tablet->partition_id();
context.tablet_schema_hash = new_tablet->schema_hash();
context.rowset_type = new_rowset_type;
+if (StorageEngine::instance()->set_default_rowset_type_to_beta()) {
Review comment:
I think we need a framework to set this rowset_type, instead of setting this
at each time the RowsetWriterContext created.
Or it is very easy to miss some places.
Maybe in `RowsetWriterContext`'s constructor?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
morningman commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247 URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353042740 ## File path: fe/src/main/java/org/apache/doris/qe/GlobalVariable.java ## @@ -56,6 +56,11 @@ @VariableMgr.VarAttr(name = "query_cache_size") private static long queryCacheSize = 1048576; +public static final String HEARTBEAT_FLAGS = "heartbeat_flags"; Review comment: `public static final String HEARTBEAT_FLAGS = "heartbeat_flags";` move this line to the above, with other variables' name declaration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman commented on a change in pull request #2364: add a control framework between FE and BE through heartbeat #2247
morningman commented on a change in pull request #2364: add a control framework
between FE and BE through heartbeat #2247
URL: https://github.com/apache/incubator-doris/pull/2364#discussion_r353038861
##
File path: be/src/olap/heartbeat_flags_helper.h
##
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+
+namespace doris {
+
+class HeartbeatFlagsHelper {
Review comment:
Add comment to this class to explain what is this class for and how to use
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r353038069
##
File path: be/src/olap/memtable.h
##
@@ -56,7 +61,13 @@ class MemTable {
RowCursorComparator _row_comparator;
std::unique_ptr _mem_tracker;
-std::unique_ptr _mem_pool;
+// This is a buffer, to hold the memory referenced by the rows that have
not
+// been inserted into the SkipList
+std::unique_ptr _tmp_mem_pool;
Review comment:
OK, I will change it to `_buffer_mem_pool`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] lingbin commented on a change in pull request #2359: Improve SkipList memory usage tracking
lingbin commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r353037328
##
File path: be/src/olap/memtable.cpp
##
@@ -61,26 +62,49 @@ int MemTable::RowCursorComparator::operator()(const char*
left, const char* righ
return compare_row(lhs_row, rhs_row);
}
-size_t MemTable::memory_usage() {
-return _mem_pool->mem_tracker()->consumption();
-}
+void MemTable::insert(const Tuple* tuple) {
+bool overwritten = false;
+if (_keys_type == KeysType::DUP_KEYS) {
+// Will insert directly, so use memory from _table_mem_pool
+_tuple_buf = _table_mem_pool->allocate(_schema_size);
+ContiguousRow row(_schema, _tuple_buf);
+_tuple_to_row(tuple, , _table_mem_pool.get());
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(!overwritten) << "Duplicate key model meet overwrite in
SkipList";
+return;
+}
-void MemTable::insert(Tuple* tuple) {
+_tuple_buf = _tmp_mem_pool->allocate(_schema_size);
ContiguousRow row(_schema, _tuple_buf);
+_tuple_to_row(tuple, , _tmp_mem_pool.get());
+
+// TODO(lingbin): Remove redundant contain check
+if (_skip_list->Contains((char*)_tuple_buf)) {
+// Will aggregate, use memory from _tmp_mem_pool
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(overwritten) << "Does not meet duplicated key in SkipList";
+} else {
+// Will insert directly, so use memory from _table_mem_pool
+_tuple_buf = _table_mem_pool->allocate(_schema_size);
+ContiguousRow dst_row(_schema, _tuple_buf);
+copy_row(_row, row, _table_mem_pool.get());
+_skip_list->Insert((char*)_tuple_buf, );
+DCHECK(!overwritten) << "Meet unexpected duplicated key in SkipList";
+}
+// Make MemPool to be reusable, but does not free its memory
+_tmp_mem_pool->clear();
+}
+
+void MemTable::_tuple_to_row(const Tuple* tuple, ContiguousRow* row, MemPool*
mem_pool) {
Review comment:
Already fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on issue #2247: add a flag mechanism for FE to control all BEs' behaviour
kangpinghuang commented on issue #2247: add a flag mechanism for FE to control all BEs' behaviour URL: https://github.com/apache/incubator-doris/issues/2247#issuecomment-561053798 set default rowset type to beta, just use the set command as following: set GLOBAL heartbeat_flags = 1; or set GLOBAL heartbeat_flags = SET_DEFAULT_ROWSET_TYPE_TO_BETA; This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang opened a new pull request #2364: add a control framework between FE and BE through heartbeat #2247
kangpinghuang opened a new pull request #2364: add a control framework between FE and BE through heartbeat #2247 URL: https://github.com/apache/incubator-doris/pull/2364 The control framework is realize through heartbeat message. Use uint64_t as flags to control different functions. Now add a flag to set the default rowset type to beta. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request #2359: Improve SkipList memory usage tracking
chaoyli commented on a change in pull request #2359: Improve SkipList memory
usage tracking
URL: https://github.com/apache/incubator-doris/pull/2359#discussion_r353034261
##
File path: be/src/olap/memtable.h
##
@@ -56,7 +61,13 @@ class MemTable {
RowCursorComparator _row_comparator;
std::unique_ptr _mem_tracker;
-std::unique_ptr _mem_pool;
+// This is a buffer, to hold the memory referenced by the rows that have
not
+// been inserted into the SkipList
+std::unique_ptr _tmp_mem_pool;
Review comment:
_buffer_pool may be better
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2319: Add Bitmap index reader
imay commented on a change in pull request #2319: Add Bitmap index reader
URL: https://github.com/apache/incubator-doris/pull/2319#discussion_r353027079
##
File path: be/src/olap/rowset/segment_v2/column_reader.cpp
##
@@ -536,17 +518,17 @@ Status DefaultValueColumnIterator::init(const
ColumnIteratorOptions& opts) {
return Status::OK();
}
-Status DefaultValueColumnIterator::next_batch(size_t* n, ColumnBlock* dst) {
+Status DefaultValueColumnIterator::next_batch(size_t* n, ColumnBlockView* dst)
{
+if (dst->is_nullable()) {
+dst->set_null_bits(*n, _is_default_value_null);
+}
+
if (_is_default_value_null) {
-for (int i = 0; i < *n; ++i) {
-dst->set_is_null(i, true);
-}
+dst->advance(*n);
} else {
for (int i = 0; i < *n; ++i) {
-memcpy(dst->mutable_cell_ptr(i), _mem_value.data(), _value_size);
-if (dst->is_nullable()) {
-dst->set_is_null(i, false);
-}
+memcpy(dst->data(), _mem_value.data(), _value_size);
+dst->advance(1);
Review comment:
OK
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman opened a new pull request #2363: Remove the 'isRestore' flag when creating table or partition
morningman opened a new pull request #2363: Remove the 'isRestore' flag when creating table or partition URL: https://github.com/apache/incubator-doris/pull/2363 'isRestore' flag is for the old version of backup and restore process, which is deprecated now. Remove it. This commit is also for making a further step to ISSUE #1723. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x
morningman merged pull request #2356: Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x URL: https://github.com/apache/incubator-doris/pull/2356 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[incubator-doris] branch master updated (875790e -> 086bb82)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-doris.git. from 875790e Remove VersionHash used to comparation in Fe (#2335) add 086bb82 Fixed a bug that Load job's state is incorrect when upgrading from 0.10.x to 0.11.x (#2356) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/http/HttpServer.java | 2 +- .../main/java/org/apache/doris/load/LoadJob.java | 1 + .../org/apache/doris/load/loadv2/JobState.java | 1 + .../java/org/apache/doris/load/loadv2/LoadJob.java | 16 +-- .../org/apache/doris/load/loadv2/LoadManager.java | 54 +- .../doris/transaction/GlobalTransactionMgr.java| 18 +++- .../org/apache/doris/http/DorisHttpTestCase.java | 2 +- 7 files changed, 86 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
