Xikui Wang has submitted this change and it was merged. Change subject: [NO ISSUE][FUN] Add more data types for external library ......................................................................
[NO ISSUE][FUN] Add more data types for external library - user model changes: no - storage format changes: no - interface changes: no Details: 1. Added more types for external library. 2. Added test case for data types. 3. Fixed minor bug where the parameter type is not trimmed in UDF. Change-Id: I6ce73e791533b0617074536e0d841242d9e0ee31 Reviewed-on: https://asterix-gerrit.ics.uci.edu/2871 Sonar-Qube: Jenkins <[email protected]> Tested-by: Jenkins <[email protected]> Contrib: Jenkins <[email protected]> Integration-Tests: Jenkins <[email protected]> Reviewed-by: Dmitry Lychagin <[email protected]> --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.1.lib.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.2.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.3.lib.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.4.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/type_validation/type_validation.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/validate-default-library/validate-default-library.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_it_sqlpp.xml A asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunction.java A asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunctionFactory.java M asterixdb/asterix-external-data/src/test/resources/library_descriptor.xml M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java M asterixdb/asterix-server/src/test/resources/integrationts/library/results/library-metadata/functionDataset/functionDataset.1.adm 13 files changed, 229 insertions(+), 7 deletions(-) Approvals: Anon. E. Moose #1000171: Jenkins: Verified; No violations found; ; Verified Dmitry Lychagin: Looks good to me, approved diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java index 0af1b9d..0eac212 100755 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java @@ -222,7 +222,7 @@ String functionType = function.getFunctionType().trim(); List<String> args = new ArrayList<>(); for (String arg : fargs) { - args.add(arg); + args.add(arg.trim()); } FunctionSignature signature = new FunctionSignature(dataverse, functionFullName, args.size()); Function f = new Function(signature, args, functionReturnType, functionDefinition, functionLanguage, diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.1.lib.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.1.lib.sqlpp new file mode 100644 index 0000000..25aff38 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.1.lib.sqlpp @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +install externallibtest testlib target/data/externallib/asterix-external-data-testlib.zip \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.2.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.2.query.sqlpp new file mode 100644 index 0000000..b09aa25 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.2.query.sqlpp @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +use externallibtest; +testlib#typeValidation(907, 9.07, "907", 9.07, true, create_point(1.0, 1.0),date("2013-01-01"), datetime("1989-09-07T12:13:14.039Z"), create_line(create_point(1.0, 1.0), create_point(2.0, 2.0)), create_circle(create_point(1.0, 1.0), 2.0), create_rectangle(create_point(1.0, 1.0), create_point(2.0, 2.0))); diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.3.lib.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.3.lib.sqlpp new file mode 100644 index 0000000..ffde186 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.3.lib.sqlpp @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +uninstall externallibtest testlib \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.4.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.4.ddl.sqlpp new file mode 100644 index 0000000..2b27030 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-library/type_validation/type_validation.4.ddl.sqlpp @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +DROP DATAVERSE externallibtest; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/type_validation/type_validation.1.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/type_validation/type_validation.1.adm new file mode 100644 index 0000000..874b506 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/type_validation/type_validation.1.adm @@ -0,0 +1 @@ +"907 9.07 \"907\" 9.07 TRUE point: { x: 1.0, y: 1.0 } \"date\": { 2013-01-01 } datetime: { 1989-09-07T12:13:14.039Z } line: { p1: point: { x: 1.0, y: 1.0 }, p2: point: { x: 2.0, y: 2.0 }} circle: { \"center\": point: { x: 1.0, y: 1.0 }, \"radius\":2.0} rectangle: { p1: point: { x: 1.0, y: 1.0 }, p2: point: { x: 2.0, y: 2.0 }}" \ No newline at end of file diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/validate-default-library/validate-default-library.1.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/validate-default-library/validate-default-library.1.adm index 9b2714a..a306a90 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/validate-default-library/validate-default-library.1.adm +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/validate-default-library/validate-default-library.1.adm @@ -8,3 +8,4 @@ { "Function": { "DataverseName": "externallibtest", "Name": "testlib#mysum", "Arity": "2", "Params": [ "AINT32", "AINT32" ], "ReturnType": "AINT32", "Definition": "org.apache.asterix.external.library.SumFactory", "Language": "JAVA", "Kind": "SCALAR", "Dependencies": [ [ ], [ ] ] } } { "Function": { "DataverseName": "externallibtest", "Name": "testlib#parseTweet", "Arity": "1", "Params": [ "TweetInputType" ], "ReturnType": "TweetOutputType", "Definition": "org.apache.asterix.external.library.ParseTweetFactory", "Language": "JAVA", "Kind": "SCALAR", "Dependencies": [ [ ], [ ] ] } } { "Function": { "DataverseName": "externallibtest", "Name": "testlib#toUpper", "Arity": "1", "Params": [ "TextType" ], "ReturnType": "TextType", "Definition": "org.apache.asterix.external.library.UpperCaseFactory", "Language": "JAVA", "Kind": "SCALAR", "Dependencies": [ [ ], [ ] ] } } +{ "Function": { "DataverseName": "externallibtest", "Name": "testlib#typeValidation", "Arity": "11", "Params": [ "AINT32", "AFLOAT", "ASTRING", "ADouble", "ABoolean", "APoint", "ADate", "ADatetime", "ALine", "ACircle", "ARectangle" ], "ReturnType": "AString", "Definition": "org.apache.asterix.external.library.TypeValidationFunctionFactory", "Language": "JAVA", "Kind": "SCALAR", "Dependencies": [ [ ], [ ] ] } } diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_it_sqlpp.xml b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_it_sqlpp.xml index 1d591e1..2121aae 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_it_sqlpp.xml +++ b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_it_sqlpp.xml @@ -24,6 +24,11 @@ <test-group name="external-library"> <test-case FilePath="external-library"> + <compilation-unit name="type_validation"> + <output-dir compare="Text">type_validation</output-dir> + </compilation-unit> + </test-case> + <test-case FilePath="external-library"> <compilation-unit name="mysum"> <output-dir compare="Text">mysum</output-dir> </compilation-unit> diff --git a/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunction.java b/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunction.java new file mode 100644 index 0000000..93bffe2 --- /dev/null +++ b/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunction.java @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.asterix.external.library; + +import org.apache.asterix.external.api.IExternalScalarFunction; +import org.apache.asterix.external.api.IFunctionHelper; +import org.apache.asterix.external.library.java.base.JBoolean; +import org.apache.asterix.external.library.java.base.JCircle; +import org.apache.asterix.external.library.java.base.JDate; +import org.apache.asterix.external.library.java.base.JDateTime; +import org.apache.asterix.external.library.java.base.JDouble; +import org.apache.asterix.external.library.java.base.JFloat; +import org.apache.asterix.external.library.java.base.JInt; +import org.apache.asterix.external.library.java.base.JLine; +import org.apache.asterix.external.library.java.base.JPoint; +import org.apache.asterix.external.library.java.base.JRectangle; +import org.apache.asterix.external.library.java.base.JString; + +public class TypeValidationFunction implements IExternalScalarFunction { + + private JString result; + + @Override + public void deinitialize() { + // no op + } + + @Override + public void evaluate(IFunctionHelper functionHelper) throws Exception { + JInt int32 = (JInt) functionHelper.getArgument(0); + JFloat floatVal = (JFloat) functionHelper.getArgument(1); + JString stringVal = (JString) functionHelper.getArgument(2); + JDouble doubleVal = (JDouble) functionHelper.getArgument(3); + JBoolean booleanVal = (JBoolean) functionHelper.getArgument(4); + JPoint pointVal = (JPoint) functionHelper.getArgument(5); + JDate dateVal = (JDate) functionHelper.getArgument(6); + JDateTime datetimeVal = (JDateTime) functionHelper.getArgument(7); + JLine lineVal = (JLine) functionHelper.getArgument(8); + JCircle circleVal = (JCircle) functionHelper.getArgument(9); + JRectangle rectangleVal = (JRectangle) functionHelper.getArgument(10); + + StringBuilder sb = new StringBuilder(); + sb.append(int32.getIAObject() + " "); + sb.append(floatVal.getIAObject() + " "); + sb.append(stringVal.getIAObject() + " "); + sb.append(doubleVal.getIAObject() + " "); + sb.append(booleanVal.getIAObject() + " "); + sb.append(pointVal.getIAObject() + " "); + sb.append(dateVal.getIAObject() + " "); + sb.append(datetimeVal.getIAObject() + " "); + sb.append(lineVal.getIAObject() + " "); + sb.append(circleVal.getIAObject() + " "); + sb.append(rectangleVal.getIAObject()); + result.setValue(sb.toString()); + functionHelper.setResult(result); + } + + @Override + public void initialize(IFunctionHelper functionHelper) throws Exception { + result = new JString(""); + } + +} diff --git a/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunctionFactory.java b/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunctionFactory.java new file mode 100644 index 0000000..167d16d --- /dev/null +++ b/asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/TypeValidationFunctionFactory.java @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.asterix.external.library; + +import org.apache.asterix.external.api.IExternalScalarFunction; +import org.apache.asterix.external.api.IFunctionFactory; + +public class TypeValidationFunctionFactory implements IFunctionFactory { + + @Override + public IExternalScalarFunction getExternalFunction() { + return new TypeValidationFunction(); + } + +} diff --git a/asterixdb/asterix-external-data/src/test/resources/library_descriptor.xml b/asterixdb/asterix-external-data/src/test/resources/library_descriptor.xml index de6a67f..45cdc35 100644 --- a/asterixdb/asterix-external-data/src/test/resources/library_descriptor.xml +++ b/asterixdb/asterix-external-data/src/test/resources/library_descriptor.xml @@ -100,6 +100,14 @@ <definition>org.apache.asterix.external.library.EchoDelayFactory </definition> </libraryFunction> + <libraryFunction> + <name>typeValidation</name> + <function_type>SCALAR</function_type> + <argument_type>AINT32, AFLOAT, ASTRING, ADouble, ABoolean, APoint, ADate, ADatetime, ALine, ACircle, ARectangle</argument_type> + <return_type>AString</return_type> + <definition>org.apache.asterix.external.library.TypeValidationFunctionFactory + </definition> + </libraryFunction> </libraryFunctions> <libraryAdapters> <libraryAdapter> diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java index c2b7bb6..01d0af7 100644 --- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java +++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java @@ -89,13 +89,29 @@ private static IAType getTypeInfo(String paramType, MetadataTransactionContext txnCtx, Function function) throws AlgebricksException { if (paramType.equalsIgnoreCase(BuiltinType.AINT32.getDisplayName())) { - return (BuiltinType.AINT32); + return BuiltinType.AINT32; } else if (paramType.equalsIgnoreCase(BuiltinType.AFLOAT.getDisplayName())) { - return (BuiltinType.AFLOAT); + return BuiltinType.AFLOAT; } else if (paramType.equalsIgnoreCase(BuiltinType.ASTRING.getDisplayName())) { - return (BuiltinType.ASTRING); + return BuiltinType.ASTRING; } else if (paramType.equalsIgnoreCase(BuiltinType.ADOUBLE.getDisplayName())) { - return (BuiltinType.ADOUBLE); + return BuiltinType.ADOUBLE; + } else if (paramType.equalsIgnoreCase(BuiltinType.ABOOLEAN.getDisplayName())) { + return BuiltinType.ABOOLEAN; + } else if (paramType.equalsIgnoreCase(BuiltinType.APOINT.getDisplayName())) { + return BuiltinType.APOINT; + } else if (paramType.equalsIgnoreCase(BuiltinType.ADATE.getDisplayName())) { + return BuiltinType.ADATE; + } else if (paramType.equalsIgnoreCase(BuiltinType.ADATETIME.getDisplayName())) { + return BuiltinType.ADATETIME; + } else if (paramType.equalsIgnoreCase(BuiltinType.APOINT3D.getDisplayName())) { + return BuiltinType.APOINT3D; + } else if (paramType.equalsIgnoreCase(BuiltinType.ALINE.getDisplayName())) { + return BuiltinType.ALINE; + } else if (paramType.equalsIgnoreCase(BuiltinType.ACIRCLE.getDisplayName())) { + return BuiltinType.ACIRCLE; + } else if (paramType.equalsIgnoreCase(BuiltinType.ARECTANGLE.getDisplayName())) { + return BuiltinType.ARECTANGLE; } else { IAType collection = getCollectionType(paramType, txnCtx, function); if (collection != null) { @@ -104,9 +120,9 @@ Datatype datatype; datatype = MetadataManager.INSTANCE.getDatatype(txnCtx, function.getDataverseName(), paramType); if (datatype == null) { - throw new MetadataException(" Type " + paramType + " not defined"); + throw new MetadataException(" Type " + paramType + " is not supported in UDF."); } - return (datatype.getDatatype()); + return datatype.getDatatype(); } } } diff --git a/asterixdb/asterix-server/src/test/resources/integrationts/library/results/library-metadata/functionDataset/functionDataset.1.adm b/asterixdb/asterix-server/src/test/resources/integrationts/library/results/library-metadata/functionDataset/functionDataset.1.adm index b1fa1d1..74b1847 100644 --- a/asterixdb/asterix-server/src/test/resources/integrationts/library/results/library-metadata/functionDataset/functionDataset.1.adm +++ b/asterixdb/asterix-server/src/test/resources/integrationts/library/results/library-metadata/functionDataset/functionDataset.1.adm @@ -8,3 +8,4 @@ { "DataverseName": "externallibtest", "Name": "testlib#mysum", "Arity": "2", "Params": [ "AINT32", "AINT32" ], "ReturnType": "AINT32", "Definition": "org.apache.asterix.external.library.SumFactory", "Language": "JAVA", "Kind": "SCALAR" } { "DataverseName": "externallibtest", "Name": "testlib#parseTweet", "Arity": "1", "Params": [ "TweetInputType" ], "ReturnType": "TweetOutputType", "Definition": "org.apache.asterix.external.library.ParseTweetFactory", "Language": "JAVA", "Kind": "SCALAR" } { "DataverseName": "externallibtest", "Name": "testlib#toUpper", "Arity": "1", "Params": [ "TextType" ], "ReturnType": "TextType", "Definition": "org.apache.asterix.external.library.UpperCaseFactory", "Language": "JAVA", "Kind": "SCALAR" } +{ "DataverseName": "externallibtest", "Name": "testlib#typeValidation", "Arity": "11", "Params": [ "AINT32", "AFLOAT", "ASTRING", "ADouble", "ABoolean", "APoint", "ADate", "ADatetime", "ALine", "ACircle", "ARectangle" ], "ReturnType": "AString", "Definition": "org.apache.asterix.external.library.TypeValidationFunctionFactory", "Language": "JAVA", "Kind": "SCALAR" } \ No newline at end of file -- To view, visit https://asterix-gerrit.ics.uci.edu/2871 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: merged Gerrit-Change-Id: I6ce73e791533b0617074536e0d841242d9e0ee31 Gerrit-PatchSet: 12 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Xikui Wang <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Dmitry Lychagin <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Till Westmann <[email protected]> Gerrit-Reviewer: Xikui Wang <[email protected]>
