[
https://issues.apache.org/jira/browse/CALCITE-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167303#comment-17167303
]
zhangchenghui edited comment on CALCITE-4144 at 7/29/20, 3:12 PM:
------------------------------------------------------------------
[~zabetak]
At present, there is only one of them in my project, and no other conditions
are added. Moreover, a custom compiler I wrote is only used for conditional
filtering. The specific code is: org.apache.calcite.
Interpreter.TableScanNode#createEnumerable.
After a customized compiler, JaninoRexCompiler is perfectly bypassed. Since
reflection is no longer used, query performance has been greatly improved. But
my compiler is only suitable for SQL in my project.
My specific implementation is as follows:
{code:java}
/**
* execute for example:
* <p>
* final Object[] current = context.values;
* final long input_value =
org.apache.calcite.runtime.SqlFunctions.toLong(current[1]);
* final String input_value0 = current[0] == null ? (String) null :
current[0].toString();
* outputValues[0] =
org.apache.calcite.runtime.SqlFunctions.toBoolean(input_value >= 1595988840000L
&& input_value <= 1595992440000L
* && (input_value0 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value0, "ZTO_TP_TitansDemo")));
* <p>
* final Object[] current = context.values;
* final long input_value =
org.apache.calcite.runtime.SqlFunctions.toLong(current[1]);
* final String input_value0 = current[0] == null ? (String) null :
current[0].toString();
* final String input_value1 = current[2] == null ? (String) null :
current[2].toString();
* final String input_value2 = current[3] == null ? (String) null :
current[3].toString();
* final boolean input_isNull2 = input_value2 == null;
* final String input_value3 = current[8] == null ? (String) null :
current[8].toString();
* outputValues[0] =
org.apache.calcite.runtime.SqlFunctions.toBoolean(input_value >= 1595988840000L
&& input_value <= 1595992440000L
* && (input_value0 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value0, "ZTO_TP_TitansDemo"))
* && (input_value1 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value1, "10.10.138.238"))
* && org.apache.calcite.runtime.SqlFunctions.toBoolean(!input_isNull2 &&
* org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "DubboService")
* || !input_isNull2 &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "HttpCall")
* || !input_isNull2 &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "DubboClient")
* || !input_isNull2 &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "URL"))
* && (input_value3 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value3, "$$TOTAL")));
* <p>
* {@link
ExtendedEnumerable#where(org.apache.calcite.linq4j.function.Predicate1)}
*
* @author [email protected]
* @since 1.0.0
*/
public class MyScalar implements Scalar {
private final RexNode rexNode;
public ZCATScalar(RexNode rexNode) {
this.rexNode = rexNode;
}
@Override
public void execute(final Context context, Object[] outputValues) {
RexNodeValue rexNodeValue = new RexNodeValue();
parseRexNode(rexNode, rexNodeValue);
// 从moveNext查询出来的 current 值
final Object[] current = context.values;
final String appId =
current[ReportConst.ColumnNumber.APPID.getNum()].toString();
final long reportTime =
org.apache.calcite.runtime.SqlFunctions.toLong(current[ReportConst.ColumnNumber.REPORTTIME.getNum()]);
final String location =
current[ReportConst.ColumnNumber.LOCATION.getNum()].toString();
final String group2 =
current[ReportConst.ColumnNumber.GROUP2.getNum()].toString();
final String metricKey =
current[ReportConst.ColumnNumber.METRICKEY.getNum()].toString();
// calcite 过滤条件
List<SqlCondition> locationSqlConditions =
rexNodeValue.getSqlConditions().get(LOCATION);
List<SqlCondition> group2SqlConditions =
rexNodeValue.getSqlConditions().get(GROUP2);
List<SqlCondition> metricKeySqlConditions =
rexNodeValue.getSqlConditions().get(METRICKEY);
outputValues[0] = SqlFunctions.toBoolean(
SqlFunctions.eq(appId, rexNodeValue.getAppId())
&& reportTime >= rexNodeValue.getStartTime() && reportTime <=
rexNodeValue.getEndTime()
&& validSqlCondition(locationSqlConditions, location)
&& validSqlCondition(group2SqlConditions, group2)
&& validSqlCondition(metricKeySqlConditions, metricKey)
);
}
@Override
public Object execute(Context context) {
final Object[] values = new Object[1];
this.execute(context, values);
return values[0];
}
private boolean validSqlCondition(List<SqlCondition> sqlConditions, String
conditionValue) {
if (sqlConditions == null) {
return true;
}
if (SqlKind.EQUALS.name().equals(sqlConditions.get(0).getOperator())) {
return sqlConditions
.stream()
.anyMatch(c -> conditionValue != null &&
SqlFunctions.eq(conditionValue, c.getValue()));
} else if
(SqlKind.NOT_EQUALS.name().equals(sqlConditions.get(0).getOperator())) {
return sqlConditions
.stream()
.allMatch(c -> conditionValue != null &&
SqlFunctions.ne(conditionValue, c.getValue()));
} else {
return false;
}
}
private RexNodeValue parseRexNode(RexNode rexNode, RexNodeValue
rexNodeValue) {
if (rexNode.isA(SqlKind.AND) || rexNode.isA(SqlKind.OR)) {
((RexCall) rexNode).getOperands().forEach(subRexNode ->
parseRexNode(subRexNode, rexNodeValue));
} else if (rexNode.isA(Arrays.asList(SqlKind.EQUALS,
SqlKind.NOT_EQUALS, SqlKind.GREATER_THAN_OR_EQUAL,
SqlKind.GREATER_THAN, SqlKind.LESS_THAN,
SqlKind.LESS_THAN_OR_EQUAL))) {
final RexCall call = (RexCall) rexNode;
RexNode left = call.getOperands().get(0);
if (left.isA(SqlKind.CAST)) {
left = ((RexCall) left).operands.get(0);
}
final RexNode right = call.getOperands().get(1);
if (left instanceof RexInputRef
&& right instanceof RexLiteral) {
final int index = ((RexInputRef) left).getIndex();
if (index == ReportConst.ColumnNumber.APPID.getNum()) { // 0
rexNodeValue.setAppId(((RexLiteral)
right).getValue2().toString());
} else if (index ==
ReportConst.ColumnNumber.REPORTTIME.getNum()) { // 1
if (call.isA(Arrays.asList(SqlKind.GREATER_THAN_OR_EQUAL,
SqlKind.GREATER_THAN))) {
rexNodeValue.setStartTime((long) ((RexLiteral)
right).getValue2());
} else if
(call.isA(Arrays.asList(SqlKind.LESS_THAN_OR_EQUAL, SqlKind.LESS_THAN))) {
rexNodeValue.setEndTime((long) ((RexLiteral)
right).getValue2());
}
} else if (index == ReportConst.ColumnNumber.LOCATION.getNum())
{ // 2
rexNodeValue.getSqlConditions().computeIfAbsent(LOCATION, k
-> new ArrayList<>());
SqlCondition sqlCondition = new
SqlCondition(call.getKind().name(), (String) ((RexLiteral) right).getValue2());
rexNodeValue.getSqlConditions().get(LOCATION).add(sqlCondition);
} else if (index == ReportConst.ColumnNumber.GROUP2.getNum()) {
// 3
rexNodeValue.getSqlConditions().computeIfAbsent(GROUP2, k
-> new ArrayList<>());
SqlCondition sqlCondition = new
SqlCondition(call.getKind().name(), (String) ((RexLiteral) right).getValue2());
rexNodeValue.getSqlConditions().get(GROUP2).add(sqlCondition);
} else if (index ==
ReportConst.ColumnNumber.METRICKEY.getNum()) { // 8
rexNodeValue.getSqlConditions().computeIfAbsent(METRICKEY,
k -> new ArrayList<>());
SqlCondition sqlCondition = new
SqlCondition(call.getKind().name(), (String) ((RexLiteral) right).getValue2());
rexNodeValue.getSqlConditions().get(METRICKEY).add(sqlCondition);
}
}
}
return rexNodeValue;
}
private static class RexNodeValue {
private String appId;
private long startTime;
private long endTime;
private Map<String, List<SqlCondition>> sqlConditions = new HashMap<>();
public String getAppId() {
return appId;
}
public void setAppId(String appId) {
this.appId = appId;
}
public long getStartTime() {
return startTime;
}
public void setStartTime(long startTime) {
this.startTime = startTime;
}
public long getEndTime() {
return endTime;
}
public void setEndTime(long endTime) {
this.endTime = endTime;
}
public Map<String, List<SqlCondition>> getSqlConditions() {
return sqlConditions;
}
public void setSqlConditions(Map<String, List<SqlCondition>>
sqlConditions) {
this.sqlConditions = sqlConditions;
}
}
}
{code}
{code:java}
public class MySimpleScalar implements Scalar {
@Override
public void execute(final Context context, Object[] outputValues) {
outputValues[0] = true;
}
@Override
public Object execute(Context context) {
final Object[] values = new Object[1];
this.execute(context, values);
return values[0];
}
}
{code}
{code:java}
public class MyCompiler implements Interpreter.ScalarCompiler {
@Override
public Scalar compile(List<RexNode> nodes, RelDataType inputRowType) {
return new MyScalar(nodes.get(0));
// return new MySimpleScalar();
}
}
{code}
org.apache.calcite.interpreter.Interpreter.CompilerImpl#CompilerImpl
{code:java}
CompilerImpl(Interpreter interpreter, RelOptCluster cluster) {
this.interpreter = interpreter;
// this.scalarCompiler = new JaninoRexCompiler(cluster.getRexBuilder());
this.scalarCompiler = new MyCompiler();
}
{code}
was (Author: zhangchenghui):
[~zabetak]
At present, there is only one of them in my project, and no other conditions
are added. Moreover, a custom compiler I wrote is only used for conditional
filtering. The specific code is: org.apache.calcite.
Interpreter.TableScanNode#createEnumerable.
After a customized compiler, JaninoRexCompiler is perfectly bypassed. Since
reflection is no longer used, query performance has been greatly improved. But
my compiler is only suitable for SQL in my project.
My specific implementation is as follows:
{code:java}
/**
* execute for example:
* <p>
* final Object[] current = context.values;
* final long input_value =
org.apache.calcite.runtime.SqlFunctions.toLong(current[1]);
* final String input_value0 = current[0] == null ? (String) null :
current[0].toString();
* outputValues[0] =
org.apache.calcite.runtime.SqlFunctions.toBoolean(input_value >= 1595988840000L
&& input_value <= 1595992440000L
* && (input_value0 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value0, "ZTO_TP_TitansDemo")));
* <p>
* final Object[] current = context.values;
* final long input_value =
org.apache.calcite.runtime.SqlFunctions.toLong(current[1]);
* final String input_value0 = current[0] == null ? (String) null :
current[0].toString();
* final String input_value1 = current[2] == null ? (String) null :
current[2].toString();
* final String input_value2 = current[3] == null ? (String) null :
current[3].toString();
* final boolean input_isNull2 = input_value2 == null;
* final String input_value3 = current[8] == null ? (String) null :
current[8].toString();
* outputValues[0] =
org.apache.calcite.runtime.SqlFunctions.toBoolean(input_value >= 1595988840000L
&& input_value <= 1595992440000L
* && (input_value0 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value0, "ZTO_TP_TitansDemo"))
* && (input_value1 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value1, "10.10.138.238"))
* && org.apache.calcite.runtime.SqlFunctions.toBoolean(!input_isNull2 &&
* org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "DubboService")
* || !input_isNull2 &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "HttpCall")
* || !input_isNull2 &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "DubboClient")
* || !input_isNull2 &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value2, "URL"))
* && (input_value3 != null &&
org.apache.calcite.runtime.SqlFunctions.eq(input_value3, "$$TOTAL")));
* <p>
* {@link
ExtendedEnumerable#where(org.apache.calcite.linq4j.function.Predicate1)}
*
* @author [email protected]
* @since 1.0.0
*/
public class MyScalar implements Scalar {
private final RexNode rexNode;
public ZCATScalar(RexNode rexNode) {
this.rexNode = rexNode;
}
@Override
public void execute(final Context context, Object[] outputValues) {
RexNodeValue rexNodeValue = new RexNodeValue();
parseRexNode(rexNode, rexNodeValue);
// 从moveNext查询出来的 current 值
final Object[] current = context.values;
final String appId =
current[ReportConst.ColumnNumber.APPID.getNum()].toString();
final long reportTime =
org.apache.calcite.runtime.SqlFunctions.toLong(current[ReportConst.ColumnNumber.REPORTTIME.getNum()]);
final String location =
current[ReportConst.ColumnNumber.LOCATION.getNum()].toString();
final String group2 =
current[ReportConst.ColumnNumber.GROUP2.getNum()].toString();
final String metricKey =
current[ReportConst.ColumnNumber.METRICKEY.getNum()].toString();
// calcite 过滤条件
List<SqlCondition> locationSqlConditions =
rexNodeValue.getSqlConditions().get(LOCATION);
List<SqlCondition> group2SqlConditions =
rexNodeValue.getSqlConditions().get(GROUP2);
List<SqlCondition> metricKeySqlConditions =
rexNodeValue.getSqlConditions().get(METRICKEY);
outputValues[0] = SqlFunctions.toBoolean(
SqlFunctions.eq(appId, rexNodeValue.getAppId())
&& reportTime >= rexNodeValue.getStartTime() && reportTime <=
rexNodeValue.getEndTime()
&& validSqlCondition(locationSqlConditions, location)
&& validSqlCondition(group2SqlConditions, group2)
&& validSqlCondition(metricKeySqlConditions, metricKey)
);
}
@Override
public Object execute(Context context) {
final Object[] values = new Object[1];
this.execute(context, values);
return values[0];
}
private boolean validSqlCondition(List<SqlCondition> sqlConditions, String
conditionValue) {
if (sqlConditions == null) {
return true;
}
if (SqlKind.EQUALS.name().equals(sqlConditions.get(0).getOperator())) {
return sqlConditions
.stream()
.anyMatch(c -> conditionValue != null &&
SqlFunctions.eq(conditionValue, c.getValue()));
} else if
(SqlKind.NOT_EQUALS.name().equals(sqlConditions.get(0).getOperator())) {
return sqlConditions
.stream()
.allMatch(c -> conditionValue != null &&
SqlFunctions.ne(conditionValue, c.getValue()));
} else {
return false;
}
}
private RexNodeValue parseRexNode(RexNode rexNode, RexNodeValue
rexNodeValue) {
if (rexNode.isA(SqlKind.AND) || rexNode.isA(SqlKind.OR)) {
((RexCall) rexNode).getOperands().forEach(subRexNode ->
parseRexNode(subRexNode, rexNodeValue));
} else if (rexNode.isA(Arrays.asList(SqlKind.EQUALS,
SqlKind.NOT_EQUALS, SqlKind.GREATER_THAN_OR_EQUAL,
SqlKind.GREATER_THAN, SqlKind.LESS_THAN,
SqlKind.LESS_THAN_OR_EQUAL))) {
final RexCall call = (RexCall) rexNode;
RexNode left = call.getOperands().get(0);
if (left.isA(SqlKind.CAST)) {
left = ((RexCall) left).operands.get(0);
}
final RexNode right = call.getOperands().get(1);
if (left instanceof RexInputRef
&& right instanceof RexLiteral) {
final int index = ((RexInputRef) left).getIndex();
if (index == ReportConst.ColumnNumber.APPID.getNum()) { // 0
rexNodeValue.setAppId(((RexLiteral)
right).getValue2().toString());
} else if (index ==
ReportConst.ColumnNumber.REPORTTIME.getNum()) { // 1
if (call.isA(Arrays.asList(SqlKind.GREATER_THAN_OR_EQUAL,
SqlKind.GREATER_THAN))) {
rexNodeValue.setStartTime((long) ((RexLiteral)
right).getValue2());
} else if
(call.isA(Arrays.asList(SqlKind.LESS_THAN_OR_EQUAL, SqlKind.LESS_THAN))) {
rexNodeValue.setEndTime((long) ((RexLiteral)
right).getValue2());
}
} else if (index == ReportConst.ColumnNumber.LOCATION.getNum())
{ // 2
rexNodeValue.getSqlConditions().computeIfAbsent(LOCATION, k
-> new ArrayList<>());
SqlCondition sqlCondition = new
SqlCondition(call.getKind().name(), (String) ((RexLiteral) right).getValue2());
rexNodeValue.getSqlConditions().get(LOCATION).add(sqlCondition);
} else if (index == ReportConst.ColumnNumber.GROUP2.getNum()) {
// 3
rexNodeValue.getSqlConditions().computeIfAbsent(GROUP2, k
-> new ArrayList<>());
SqlCondition sqlCondition = new
SqlCondition(call.getKind().name(), (String) ((RexLiteral) right).getValue2());
rexNodeValue.getSqlConditions().get(GROUP2).add(sqlCondition);
} else if (index ==
ReportConst.ColumnNumber.METRICKEY.getNum()) { // 8
rexNodeValue.getSqlConditions().computeIfAbsent(METRICKEY,
k -> new ArrayList<>());
SqlCondition sqlCondition = new
SqlCondition(call.getKind().name(), (String) ((RexLiteral) right).getValue2());
rexNodeValue.getSqlConditions().get(METRICKEY).add(sqlCondition);
}
}
}
return rexNodeValue;
}
private static class RexNodeValue {
private String appId;
private long startTime;
private long endTime;
private Map<String, List<SqlCondition>> sqlConditions = new HashMap<>();
public String getAppId() {
return appId;
}
public void setAppId(String appId) {
this.appId = appId;
}
public long getStartTime() {
return startTime;
}
public void setStartTime(long startTime) {
this.startTime = startTime;
}
public long getEndTime() {
return endTime;
}
public void setEndTime(long endTime) {
this.endTime = endTime;
}
public Map<String, List<SqlCondition>> getSqlConditions() {
return sqlConditions;
}
public void setSqlConditions(Map<String, List<SqlCondition>>
sqlConditions) {
this.sqlConditions = sqlConditions;
}
}
}
{code}
{code:java}
public class MySimpleScalar implements Scalar {
@Override
public void execute(final Context context, Object[] outputValues) {
outputValues[0] = true;
}
@Override
public Object execute(Context context) {
final Object[] values = new Object[1];
this.execute(context, values);
return values[0];
}
}
{code}
{code:java}
public class MyCompiler implements Interpreter.ScalarCompiler {
@Override
public Scalar compile(List<RexNode> nodes, RelDataType inputRowType) {
return new MyScalar(nodes.get(0));
// return new MySimpleScalar(nodes.get(0));
}
}
{code}
org.apache.calcite.interpreter.Interpreter.CompilerImpl#CompilerImpl
{code:java}
CompilerImpl(Interpreter interpreter, RelOptCluster cluster) {
this.interpreter = interpreter;
// this.scalarCompiler = new JaninoRexCompiler(cluster.getRexBuilder());
this.scalarCompiler = new MyCompiler();
}
{code}
> Reduce code generation and class loading overhead when getScalar in
> JaninoRexCompiler.
> --------------------------------------------------------------------------------------
>
> Key: CALCITE-4144
> URL: https://issues.apache.org/jira/browse/CALCITE-4144
> Project: Calcite
> Issue Type: Improvement
> Components: core
> Affects Versions: 1.21.0
> Environment: version: calcite-core 1.21
> model: filterableTable
> Reporter: zhangchenghui
> Priority: Major
> Labels: cache, scalar
> Fix For: 1.25.0
>
> Attachments: image-2020-07-27-22-44-44-455.png,
> image-2020-07-27-22-45-25-350.png, image-2020-07-27-22-45-54-346.png,
> image-2020-07-27-22-46-18-306.png, image-2020-07-29-13-48-53-028.png
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> I used the FilterableTable mode in the project, but I found that the query
> was particularly slow. I used the JProfile tool to troubleshoot the thread
> time-consuming place, and then through the debug, I found that each request
> took two places:
> 1、org.apache.calcite.adapter.enumerable.EnumerableInterpretable#getBindable
> !image-2020-07-27-22-44-44-455.png!
> This place optimizes the cache settings by setting the cache size.
> 2、org.apache.calcite.interpreter.JaninoRexCompiler#baz
> !image-2020-07-27-22-45-25-350.png!
> But this place is not cached, and a new expression string is used every time
> to create it through reflection.
> JProfile tool time consumption:
> !image-2020-07-27-22-45-54-346.png!
> I originally wanted to add a layer of cache here, but found that the
> expressions generated each time are different, as follows:
> !image-2020-07-27-22-46-18-306.png!
> So you can't use the getBindable method to directly add cache.
> Is there anything you can optimize here? For example, can you generate a
> template class in advance, and then generate different objects through
> different parameter values?
> Bring tea to the boss!
> Looking forward to your reply!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
