[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2919 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91114685 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/CodeGenUtils.scala --- @@ -189,6 +188,16 @@ object CodeGenUtils { throw new CodeGenException("Interval expression type expected.") } + def requireArray(genExpr: GeneratedExpression) = +if (!TypeCheckUtils.isArray(genExpr.resultType)) { + throw new CodeGenException("Array expression type expected.") +} + + def requireInteger(genExpr: GeneratedExpression) = +if (!TypeCheckUtils.isInteger(genExpr.resultType)) { + throw new CodeGenException("Array expression type expected.") --- End diff -- `Array` -> `Integer` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91124107 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/plan/schema/ArrayRelDataType.scala --- @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.table.plan.schema + +import org.apache.calcite.rel.`type`.RelDataType +import org.apache.calcite.sql.`type`.ArraySqlType +import org.apache.flink.api.common.typeinfo.TypeInformation + +/** + * Flink distinguishes between primitive arrays (int[], double[], ...) and + * object arrays (Integer[], MyPojo[], ...). This custom type considers the two cases. --- End diff -- `considers` -> `supports` or `covers`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91126562 --- Diff: tools/maven/scalastyle-config.xml --- @@ -68,11 +68,11 @@ - --- End diff -- Can we use a custom scalastyle for `flink-table` instead of modifying the global one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91123689 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/expressions/array.scala --- @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.table.expressions + +import org.apache.calcite.rex.RexNode +import org.apache.calcite.sql.fun.SqlStdOperatorTable +import org.apache.calcite.tools.RelBuilder +import org.apache.flink.api.common.typeinfo.BasicTypeInfo.INT_TYPE_INFO +import org.apache.flink.api.common.typeinfo.{BasicTypeInfo, PrimitiveArrayTypeInfo, TypeInformation} +import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo +import org.apache.flink.api.table.FlinkRelBuilder +import org.apache.flink.api.table.validate.{ValidationFailure, ValidationResult, ValidationSuccess} + +import scala.collection.JavaConverters._ + +case class ArrayConstructor(elements: Seq[Expression]) extends Expression { + + override private[flink] def children: Seq[Expression] = elements + + override private[flink] def toRexNode(implicit relBuilder: RelBuilder): RexNode = { +val relDataType = relBuilder + .asInstanceOf[FlinkRelBuilder] + .getTypeFactory + .createTypeFromTypeInfo(resultType) +val values = elements.map(_.toRexNode).toList.asJava +relBuilder + .getRexBuilder + .makeCall(relDataType, SqlStdOperatorTable.ARRAY_VALUE_CONSTRUCTOR, values) + } + + override def toString = s"array(${elements.mkString(", ")})" + + override private[flink] def resultType = ObjectArrayTypeInfo.getInfoFor(elements.head.resultType) + + override private[flink] def validateInput(): ValidationResult = { +if (elements.isEmpty) { + return ValidationFailure("Empty arrays are not supported yet.") +} +val elementType = elements.head.resultType +if (!elements.forall(_.resultType == elementType)) { + ValidationFailure("Not all elements of the array have the same type.") +} else { + ValidationSuccess +} + } +} + +case class ArrayElementAt(array: Expression, index: Expression) extends Expression { + + override private[flink] def children: Seq[Expression] = Seq(array, index) + + override private[flink] def toRexNode(implicit relBuilder: RelBuilder): RexNode = { +relBuilder + .getRexBuilder + .makeCall(SqlStdOperatorTable.ITEM, array.toRexNode, index.toRexNode) + } + + override def toString = s"($array).at($index)" + + override private[flink] def resultType = array.resultType match { +case oati: ObjectArrayTypeInfo[_, _] => oati.getComponentInfo +case pati: PrimitiveArrayTypeInfo[_] => pati.getComponentType + } + + override private[flink] def validateInput(): ValidationResult = { --- End diff -- Does it make sense to check that the index is `> 0` if it is a literal? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91124734 --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/table/expressions/ArrayTypeTest.scala --- @@ -0,0 +1,342 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.table.expressions + +import java.sql.Date + +import org.apache.flink.api.common.typeinfo.{PrimitiveArrayTypeInfo, TypeInformation} +import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo +import org.apache.flink.api.scala.table._ +import org.apache.flink.api.table.expressions.utils.ExpressionTestBase +import org.apache.flink.api.table.typeutils.RowTypeInfo +import org.apache.flink.api.table.{Row, Types, ValidationException} +import org.junit.Test + +class ArrayTypeTest extends ExpressionTestBase { + + @Test(expected = classOf[ValidationException]) + def testEmptyArraySql(): Unit = { +testSqlApi("ARRAY[]", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testEmptyArrayTableApi(): Unit = { +testTableApi("FAIL", "array()", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testNullArraySql(): Unit = { +testSqlApi("ARRAY[NULL]", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testDifferentTypesArraySql(): Unit = { +testSqlApi("ARRAY[1, TRUE]", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testDifferentTypesArrayTableApi(): Unit = { +testTableApi("FAIL", "array(1, true)", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testUnsupportedComparison(): Unit = { +testAllApis( + 'f2 <= 'f5.at(1), + "f2 <= f5.at(1)", + "f2 <= f5[1]", + "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testElementNonArray(): Unit = { --- End diff -- test other APIs as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91110730 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/scala/table/expressionDsl.scala --- @@ -454,6 +455,29 @@ trait ImplicitExpressionOperations { * into a flat representation where every subtype is a separate field. */ def flatten() = Flattening(expr) + + /** +* Accesses the element of an array based on an index (starting at 1). +* +* @param index position of the element (starting at 1) +* @return value of the element +*/ + def at(index: Expression) = ArrayElementAt(expr, index) + + /** +* Returns the number of elements of an array. +* +* @return number of element --- End diff -- `element` -> `elements` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r9043 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/scala/table/expressionDsl.scala --- @@ -454,6 +455,29 @@ trait ImplicitExpressionOperations { * into a flat representation where every subtype is a separate field. */ def flatten() = Flattening(expr) + + /** +* Accesses the element of an array based on an index (starting at 1). +* +* @param index position of the element (starting at 1) +* @return value of the element +*/ + def at(index: Expression) = ArrayElementAt(expr, index) + + /** +* Returns the number of elements of an array. +* +* @return number of element +*/ + def cardinality() = ArrayCardinality(expr) + + /** +* Returns the sole element of an array. Returns null if the collection is empty. +* Throws an exception if the array has more than one element. +* +* @return first element of the array --- End diff -- `the only element of an array with a single entry`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91109035 --- Diff: docs/dev/table_api.md --- @@ -2577,6 +2628,50 @@ COMPOSITE.get(INT) + + +{% highlight scala %} +ARRAY.at(INT) +{% endhighlight %} + + +Returns the element at a particular location in an array. The index starts at 1. --- End diff -- the comments in the Java section probably apply to Scala as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91106487 --- Diff: docs/dev/table_api.md --- @@ -1470,7 +1470,14 @@ The Table API is built on top of Flink's DataSet and DataStream API. Internally, | `Types.INTERVAL_MONTHS`| `INTERVAL YEAR TO MONTH`| `java.lang.Integer`| | `Types.INTERVAL_MILLIS`| `INTERVAL DAY TO SECOND(3)` | `java.lang.Long` | -Advanced types such as generic types, composite types (e.g. POJOs or Tuples), and arrays can be fields of a row. Generic types and arrays are treated as a black box within Table API and SQL yet. Composite types, however, are fully supported types where fields of a composite type can be accessed using the `.get()` operator in Table API and dot operator (e.g. `MyTable.pojoColumn.myField`) in SQL. Composite types can also be flattened using `.flatten()` in Table API or `MyTable.pojoColumn.*` in SQL. + +Advanced types such as generic types, composite types (e.g. POJOs or Tuples), and array types (object or primitive arrays) can be fields of a row. + +Generic types are treated as a black box within Table API and SQL yet. + +Composite types, however, are fully supported types where fields of a composite type can be accessed using the `.get()` operator in Table API and dot operator (e.g. `MyTable.pojoColumn.myField`) in SQL. Composite types can also be flattened using `.flatten()` in Table API or `MyTable.pojoColumn.*` in SQL. + +Array types can be access using the `myArray.at(1)` operator in Table API and `myArray[1]` operator in SQL. Array literals can be created using `array(1, 2, 3)` in Table API and `ARRAY[1, 2, 3]` in SQL. --- End diff -- `can be access` -> `can be accessed` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91110871 --- Diff: flink-libraries/flink-table/src/main/scala/org/apache/flink/api/scala/table/expressionDsl.scala --- @@ -454,6 +455,29 @@ trait ImplicitExpressionOperations { * into a flat representation where every subtype is a separate field. */ def flatten() = Flattening(expr) + + /** +* Accesses the element of an array based on an index (starting at 1). +* +* @param index position of the element (starting at 1) +* @return value of the element +*/ + def at(index: Expression) = ArrayElementAt(expr, index) + + /** +* Returns the number of elements of an array. +* +* @return number of element +*/ + def cardinality() = ArrayCardinality(expr) + + /** +* Returns the sole element of an array. Returns null if the collection is empty. --- End diff -- `of an array` -> `of an array with a single element.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91124760 --- Diff: flink-libraries/flink-table/src/test/scala/org/apache/flink/api/table/expressions/ArrayTypeTest.scala --- @@ -0,0 +1,342 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.table.expressions + +import java.sql.Date + +import org.apache.flink.api.common.typeinfo.{PrimitiveArrayTypeInfo, TypeInformation} +import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo +import org.apache.flink.api.scala.table._ +import org.apache.flink.api.table.expressions.utils.ExpressionTestBase +import org.apache.flink.api.table.typeutils.RowTypeInfo +import org.apache.flink.api.table.{Row, Types, ValidationException} +import org.junit.Test + +class ArrayTypeTest extends ExpressionTestBase { + + @Test(expected = classOf[ValidationException]) + def testEmptyArraySql(): Unit = { +testSqlApi("ARRAY[]", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testEmptyArrayTableApi(): Unit = { +testTableApi("FAIL", "array()", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testNullArraySql(): Unit = { +testSqlApi("ARRAY[NULL]", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testDifferentTypesArraySql(): Unit = { +testSqlApi("ARRAY[1, TRUE]", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testDifferentTypesArrayTableApi(): Unit = { +testTableApi("FAIL", "array(1, true)", "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testUnsupportedComparison(): Unit = { +testAllApis( + 'f2 <= 'f5.at(1), + "f2 <= f5.at(1)", + "f2 <= f5[1]", + "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testElementNonArray(): Unit = { +testTableApi( + 'f0.element(), + "FAIL", + "FAIL") + } + + @Test(expected = classOf[ValidationException]) + def testCardinalityOnNonArray(): Unit = { --- End diff -- test other APIs as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91107452 --- Diff: docs/dev/table_api.md --- @@ -2027,6 +2034,50 @@ COMPOSITE.get(INT) + + +{% highlight java %} +ARRAY.at(INT) +{% endhighlight %} + + +Returns the element at a particular location in an array. The index starts at 1. --- End diff -- `location` -> `position`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91108867 --- Diff: docs/dev/table_api.md --- @@ -2027,6 +2034,50 @@ COMPOSITE.get(INT) + + +{% highlight java %} +ARRAY.at(INT) +{% endhighlight %} + + +Returns the element at a particular location in an array. The index starts at 1. + + + + + +{% highlight java %} +array(ANY [, ANY ]*) +{% endhighlight %} + + +Creates an array from a list of values. The array will be an array of objects (not primitives). + + + + + +{% highlight java %} +ARRAY.cardinality() +{% endhighlight %} + + +Returns the number of elements of an array. + + + + + +{% highlight scala %} +ARRAY.element() +{% endhighlight %} + + +Returns the sole element of an array. Returns null if the collection is empty. Throws an exception if the array has more than one element. --- End diff -- I think "sole element" in this context is not very clear. I'd rephrase it to something like: "Returns the sole element of an array with a single element." `collection` -> `array`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/2919#discussion_r91109092 --- Diff: docs/dev/table_api.md --- @@ -3378,9 +3472,32 @@ ROW (value [, value]* ) Creates a row from a list of values. +--> + + + +{% highlight text %} +array â[â index â]â +{% endhighlight %} + + +Returns the element at a particular location in an array. The index starts at 1. --- End diff -- `location` -> `position` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2919: [FLINK-4554] [table] Add support for array types
GitHub user twalthr opened a pull request: https://github.com/apache/flink/pull/2919 [FLINK-4554] [table] Add support for array types This PR adds array support for the Table API and SQL. It adds support for both primitive arrays and object arrays. It adds literals, element access, comparison and the functions `ELEMENT` and `CARDINALITY`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/twalthr/flink FLINK-4554 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2919.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2919 commit 0a93866b9e2510f07f818c3ac34db439c5b52b42 Author: twalthrDate: 2016-09-23T14:44:42Z [FLINK-4554] [table] Add support for array types --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---