[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r429511368
##
File path:
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
##
@@ -593,29 +596,77 @@ private static Expression toExpression(SqlNode node) {
// Move on to process default logic.
}
default:
-SqlBasicCall funcSqlNode = (SqlBasicCall) node;
-String funcName = funcSqlNode.getOperator().getKind().name();
-if (funcSqlNode.getOperator().getKind() == SqlKind.OTHER_FUNCTION) {
- funcName = funcSqlNode.getOperator().getName();
+return evaluateFunctionExpression((SqlBasicCall) node);
+}
+ }
+
+ private static String extractFunctionName(SqlBasicCall funcSqlNode) {
+String funcName = funcSqlNode.getOperator().getKind().name();
+if (funcSqlNode.getOperator().getKind() == SqlKind.OTHER_FUNCTION) {
+ funcName = funcSqlNode.getOperator().getName();
+}
+if (funcName.equalsIgnoreCase(SqlKind.COUNT.toString()) &&
(funcSqlNode.getFunctionQuantifier() != null)
Review comment:
this is a special handling for the case of `COUNT(DISTINCT A)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r429511102
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/function/DateTimeFunctions.java
##
@@ -216,4 +217,18 @@ static String toDateTime(Long millis, String pattern) {
static Long fromDateTime(String dateTimeString, String pattern) {
return
DateTimePatternHandler.parseDateTimeStringToEpochMillis(dateTimeString,
pattern);
}
+
+ /**
+ * Return current time as epoch millis
+ */
+ static Long now() {
+return System.currentTimeMillis();
+ }
+
+ /**
+ * Return epoch millis value based on a given date time string and it's
corresponding format.
+ */
+ public static Long formatDatetime(String input, String format) {
Review comment:
will delete
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r427744783
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/function/ScalarFunctionType.java
##
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import java.util.HashMap;
+import java.util.HashSet;
+
+
+public enum ScalarFunctionType {
+
+ NOW("now"), FORMAT_DATETIME("format_datetime");
+
+ private final String _name;
+
+ static HashMap _scalarFunctions = new
HashMap<>();
+
+ static {
+ScalarFunctionType[] values = ScalarFunctionType.values();
+for (ScalarFunctionType value : values) {
Review comment:
Removed these code path for now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r427744550
##
File path:
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
##
@@ -615,7 +618,47 @@ private static Expression toExpression(SqlNode node) {
funcExpr.getFunctionCall().addToOperands(toExpression(child));
}
}
+
+if (FunctionDefinitionRegistry.isScalarFunc(funcName) &&
isCompileTimeEvaluationPossible(funcExpr)) {
+
+ ScalarFunctionType scalarFunctionType =
ScalarFunctionType.getScalarFunctionType(funcName);
+ switch (scalarFunctionType) {
+case NOW:
+ funcExpr =
RequestUtils.getLiteralExpression(System.currentTimeMillis());
+ break;
+case FORMAT_DATETIME:
+ //DATETIME_FORMAT ('2020-01-01', '-MM-dd')
+ String input =
funcExpr.getFunctionCall().getOperands().get(0).getLiteral().getStringValue();
+ String format =
funcExpr.getFunctionCall().getOperands().get(1).getLiteral().getStringValue();
+ long output =
DateTimeFormat.forPattern(format).parseMillis(input);
+ funcExpr = RequestUtils.getLiteralExpression(output);
Review comment:
Added support for eval function of functions as long as all the
recursive params are literal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r427596245
##
File path:
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
##
@@ -615,7 +620,44 @@ private static Expression toExpression(SqlNode node) {
funcExpr.getFunctionCall().addToOperands(toExpression(child));
}
}
+
+if (FunctionRegistry.containsFunctionByName(funcName) &&
isCompileTimeEvaluationPossible(funcExpr)) {
+ int functionOperandsLength = funcSqlNode.getOperands().length;
+ FunctionInfo functionInfo =
FunctionRegistry.getFunctionByName(funcName);
+ Object[] arguments = new Object[functionOperandsLength];
+ for (int i = 0; i < functionOperandsLength; i++) {
+SqlLiteral argSqlNode = (SqlLiteral) funcSqlNode.getOperands()[i];
+arguments[i] = argSqlNode.toValue();
+ }
+ try {
+FunctionInvoker invoker = new FunctionInvoker(functionInfo);
+funcExpr =
RequestUtils.getLiteralExpression(invoker.process(arguments).toString());
+ } catch (Exception e) {
+throw new SqlCompilationException("Unsupported Scalar function - "
+ funcName);
+ }
+}
return funcExpr;
}
}
+
+ /**
+ * Utility method to check if the function can be evaluated during the query
compilation phae
+ * @param funcExpr
+ * @return true if all arguments are literals
+ */
+ private static boolean isCompileTimeEvaluationPossible(Expression funcExpr) {
Review comment:
One thing here is that the function could be used in transform functions
in query/ingestion field conversion. Ideally we should be able to evaluate any
transform function with literal here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r426983997
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/function/ScalarFunctionType.java
##
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import java.util.HashMap;
+import java.util.HashSet;
+
+
+public enum ScalarFunctionType {
+
+ NOW("now"), FORMAT_DATETIME("format_datetime");
+
+ private final String _name;
+
+ static HashMap _scalarFunctions = new
HashMap<>();
+
+ static {
+ScalarFunctionType[] values = ScalarFunctionType.values();
+for (ScalarFunctionType value : values) {
Review comment:
I somehow feel, we should still pre-register those functions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r426983585
##
File path:
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
##
@@ -615,7 +618,47 @@ private static Expression toExpression(SqlNode node) {
funcExpr.getFunctionCall().addToOperands(toExpression(child));
}
}
+
+if (FunctionDefinitionRegistry.isScalarFunc(funcName) &&
isCompileTimeEvaluationPossible(funcExpr)) {
+
+ ScalarFunctionType scalarFunctionType =
ScalarFunctionType.getScalarFunctionType(funcName);
+ switch (scalarFunctionType) {
+case NOW:
Review comment:
I think we will need to move to function registry/invoker model
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r426983585
##
File path:
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
##
@@ -615,7 +618,47 @@ private static Expression toExpression(SqlNode node) {
funcExpr.getFunctionCall().addToOperands(toExpression(child));
}
}
+
+if (FunctionDefinitionRegistry.isScalarFunc(funcName) &&
isCompileTimeEvaluationPossible(funcExpr)) {
+
+ ScalarFunctionType scalarFunctionType =
ScalarFunctionType.getScalarFunctionType(funcName);
+ switch (scalarFunctionType) {
+case NOW:
Review comment:
I think we will move to function registry/invoker model
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r426983294
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/function/ScalarFunctionType.java
##
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.function;
+
+import java.util.HashMap;
+import java.util.HashSet;
+
+
+public enum ScalarFunctionType {
+
+ NOW("now"), FORMAT_DATETIME("format_datetime");
+
+ private final String _name;
+
+ static HashMap _scalarFunctions = new
HashMap<>();
+
+ static {
+ScalarFunctionType[] values = ScalarFunctionType.values();
+for (ScalarFunctionType value : values) {
+ String upperCaseFunctionName = value.getName().toUpperCase();
+ _scalarFunctions.put(upperCaseFunctionName, value);
+ _scalarFunctions.put(upperCaseFunctionName.replace("_", ""), value);
Review comment:
e.g. presto use `_` in function names. It provides more robustness from
user perspective.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5406: Adding support to execute functions during query compilation
fx19880617 commented on a change in pull request #5406:
URL: https://github.com/apache/incubator-pinot/pull/5406#discussion_r426428899
##
File path:
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
##
@@ -615,7 +618,47 @@ private static Expression toExpression(SqlNode node) {
funcExpr.getFunctionCall().addToOperands(toExpression(child));
}
}
+
+if (FunctionDefinitionRegistry.isScalarFunc(funcName) &&
isCompileTimeEvaluationPossible(funcExpr)) {
+
+ ScalarFunctionType scalarFunctionType =
ScalarFunctionType.getScalarFunctionType(funcName);
+ switch (scalarFunctionType) {
+case NOW:
+ funcExpr =
RequestUtils.getLiteralExpression(System.currentTimeMillis());
+ break;
+case FORMAT_DATETIME:
+ //DATETIME_FORMAT ('2020-01-01', '-MM-dd')
+ String input =
funcExpr.getFunctionCall().getOperands().get(0).getLiteral().getStringValue();
+ String format =
funcExpr.getFunctionCall().getOperands().get(1).getLiteral().getStringValue();
+ long output =
DateTimeFormat.forPattern(format).parseMillis(input);
+ funcExpr = RequestUtils.getLiteralExpression(output);
Review comment:
better to implement method `eval` for each function which take operands
list as argument which could be either literal or functions.
Then this logic will just be
```funcExpr =
getScalarFunction(scalarFunctionType).eval(funcExpr.getFunctionCall().getOperands())```
##
File path:
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
##
@@ -615,7 +618,47 @@ private static Expression toExpression(SqlNode node) {
funcExpr.getFunctionCall().addToOperands(toExpression(child));
}
}
+
+if (FunctionDefinitionRegistry.isScalarFunc(funcName) &&
isCompileTimeEvaluationPossible(funcExpr)) {
+
+ ScalarFunctionType scalarFunctionType =
ScalarFunctionType.getScalarFunctionType(funcName);
+ switch (scalarFunctionType) {
+case NOW:
+ funcExpr =
RequestUtils.getLiteralExpression(System.currentTimeMillis());
+ break;
+case FORMAT_DATETIME:
+ //DATETIME_FORMAT ('2020-01-01', '-MM-dd')
+ String input =
funcExpr.getFunctionCall().getOperands().get(0).getLiteral().getStringValue();
+ String format =
funcExpr.getFunctionCall().getOperands().get(1).getLiteral().getStringValue();
+ long output =
DateTimeFormat.forPattern(format).parseMillis(input);
+ funcExpr = RequestUtils.getLiteralExpression(output);
+ break;
+default:
+ //no change, let the expression be handled during execution phase
+ }
+}
+
return funcExpr;
}
}
+
+ /**
+ * Utility method to check if the function can be evaluated during the query
compilation phae
+ * @param funcExpr
+ * @return true if all arguments are literals
+ */
+ private static boolean isCompileTimeEvaluationPossible(Expression funcExpr) {
+
+boolean compileTimeEvaluationPossible = true;
+Function functionCall = funcExpr.getFunctionCall();
+if(functionCall.getOperandsSize() > 0) {
+ for (Expression expression : functionCall.getOperands()) {
+if (expression.getType() != ExpressionType.LITERAL) {
Review comment:
this should be recursively evaluated, technically a function of function
of literal should still be true here.
E.g. `format_time(now(), '-MM-dd')`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
