[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize
the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r380154237
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:
+ * SELECT columns[4], columns[8]
+ * SELECT a.b, a.c
+ * SELECT columns, columns[1]
+ * SELECT a, a.b
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ *
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ *
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * Compatibility Rules
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ *
+ *
+ * TypeConsistent with
+ * Non-repeated MAP
+ * {@code a+} {@code a.b+}
+ * Repeated MAP
+ * {@code a+} {@code a.b+} {@code a[n].b+}>
+ * Non-repeated Scalar
+ * {@code a}
+ * Repeated Scalar
+ * {@code a} {@code a[n]}
+ * Non-repeated DICT
+ * {@code a} {@code a['key']}
+ * Repeated DICT
+ * {@code a} {@code a[n]} {@code a['key']} {@code
a[n]['key']}
Review comment:
Checked whether `m.a` is supported for `MAP` arrays: it looks like this is
not supported in Drill.
For a json file `file.json`
```
{"sa": [{"a": 1}, {"a": 2}, {"a": 3}]}
{"sa": [{"a": 1}]}
```
following query ``select t.sa.a kv from dfs.`file.json` t`` produces two
rows with `null` value each. (Should have returned an error instead?)
In case when types are known during planning, e.g. in case when querying
Hive table, there is following validation: `VALIDATION ERROR: From line 1,
column 27 to line 1, column 28: Cannot apply 'ITEM' to arguments of type
'ITEM(, )'. Supported form(s): []
[]`
(used following test in `TestHiveStructs.java`:
```
@Test
public void strWithArr2ByIdxP0111() throws Exception {
HiveTestUtilities.assertNativeScanUsed(queryBuilder(), "struct_tbl_p");
testBuilder()
.sqlQuery("SELECT rid, t.str_wa_2.fa.sn p0 FROM hive.struct_tbl_p t")
.unOrdered()
.baselineColumns("rid", "p0")
.expectsEmptyResultSet()
.go();
}
```
)
However, such behavior is present in Hive, but for repeated (Drill's) `MAP`
only (but not for repeated `DICT`), IIRC.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser URL: https://github.com/apache/drill/pull/1974#discussion_r378763948 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java ## @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.physical.resultSet.project; + +/** + * Plan-time properties of a requested column.Represents + * a consolidated view of the set of references to a column. + * For example, the project list might contain: + * SELECT columns[4], columns[8] + * SELECT a.b, a.c + * SELECT columns, columns[1] + * SELECT a, a.b + * In each case, the same column is referenced in different + * forms which are consolidated into this abstraction. + * + * The resulting information is a "pattern": a form of reference + * which which a concrete type can be compatible or not. The project Review comment: ```suggestion * with which a concrete type can be compatible or not. The project ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize
the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377035722
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/TestFileMetadataColumnParser.java
##
@@ -236,14 +199,12 @@ public void testLegacyWildcardAndFileMetadata() {
* As above, but include implicit columns before and after the
* wildcard.
*/
-
@Test
public void testLegacyWildcardAndFileMetadataMixed() {
Path filePath = new Path("hdfs:///w/x/y/z.csv");
FileMetadataOptions options = standardOptions(filePath);
options.useLegacyWildcardExpansion(true);
-options.useLegacyExpansionLocation(false);
-FileMetadataManager metadataManager = new FileMetadataManager(
+ FileMetadataManager metadataManager = new FileMetadataManager(
Review comment:
```suggestion
FileMetadataManager metadataManager = new FileMetadataManager(
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser URL: https://github.com/apache/drill/pull/1974#discussion_r378763444 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java ## @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.physical.resultSet.project; + +/** + * Plan-time properties of a requested column.Represents Review comment: ```suggestion * Plan-time properties of a requested column. Represents ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize
the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377021700
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/Qualifier.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import
org.apache.drill.exec.physical.resultSet.project.RequestedTuple.TupleProjectionType;
+
+/**
+ * Represents one level of qualifier for a column. Analogous to
+ * a {@code SchemaPath}, but represents the result of coalescing
+ * multiple occurrences of the same column.
+ */
+public class Qualifier implements QualifierContainer {
+ /**
+ * Marker to indicate that that a) the item is an
+ * array, and b) that all indexes are to be projected.
+ * Used when seeing both a and a[x].
+ */
+ private static final Set ALL_INDEXES = new HashSet<>();
+
+ private Set indexes;
+ private RequestedTuple members;
+ private Qualifier child;
+
+ @Override
+ public Qualifier qualifier() { return child; }
+
+ @Override
+ public Qualifier requireQualifier() {
+if (child == null) {
+ child = new Qualifier();
+}
+return child;
+ }
+
+ public boolean isArray() {
+return indexes != null;
+ }
+
+ public boolean hasIndexes() {
+return isArray() && indexes != ALL_INDEXES;
+ }
+
+ public boolean hasIndex(int index) {
+return hasIndexes() && indexes.contains(index);
+ }
+
+ public int maxIndex() {
+if (! hasIndexes()) {
+ return 0;
Review comment:
Is it safe to have `0` instead of `-1`, for example? Does it imply that
there's always `a[0]` selected? (I'm very likely getting this wrong.)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize
the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r378772986
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:
+ * SELECT columns[4], columns[8]
+ * SELECT a.b, a.c
+ * SELECT columns, columns[1]
+ * SELECT a, a.b
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ *
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ *
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * Compatibility Rules
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ *
+ *
+ * TypeConsistent with
+ * Non-repeated MAP
+ * {@code a+} {@code a.b+}
+ * Repeated MAP
+ * {@code a+} {@code a.b+} {@code a[n].b+}>
+ * Non-repeated Scalar
+ * {@code a}
+ * Repeated Scalar
+ * {@code a} {@code a[n]}
+ * Non-repeated DICT
+ * {@code a} {@code a['key']}
+ * Repeated DICT
+ * {@code a} {@code a[n]} {@code a['key']} {@code
a[n]['key']}
Review comment:
I am not sure whether `a['key']` pattern should be supported for `Repeated
DICT` columns - maybe an UDF, e.g. `ARRAY dict_values(DICT)`, suits
better for this purpose? As it provides clearer intentions and avoids mistakes.
What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize
the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377007494
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/projSet/ProjectionChecker.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.impl.scan.project.projSet;
+
+import org.apache.drill.common.exceptions.CustomErrorContext;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.physical.resultSet.project.RequestedColumn;
+import org.apache.drill.exec.physical.resultSet.project.RequestedTuple;
+import org.apache.drill.exec.record.metadata.ColumnMetadata;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class ProjectionChecker {
+ private static final Logger logger =
LoggerFactory.getLogger(ProjectionChecker.class);
+
+ private ProjectionChecker() { }
+
+ public static boolean isConsistent(RequestedTuple tuple, ColumnMetadata
readCol) {
+if (tuple == null || !tuple.isProjected(readCol.name())) {
+ return true;
+}
+// If the column is projected, it may be projected implicitly.
+// Only check explicit projection.
+RequestedColumn col = tuple.get(readCol.name());
+if (col == null) {
+ return true;
+} else {
+ return isConsistent(col, readCol);
+}
+ }
+
+ /**
+ * Does a superficial check of projection type against the given column.
+ * Does not handle subtleties such as DICT key types, actual types
+ * in a UNION, etc.
+ */
Review comment:
Can you add `@return` here, please?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser URL: https://github.com/apache/drill/pull/1974#discussion_r377031167 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedTuple.java ## @@ -19,13 +19,10 @@ import java.util.List; -import org.apache.drill.common.expression.PathSegment; - /** * Represents the set of columns projected for a tuple (row or map.) - * The projected columns might themselves be columns, so returns a - * projection set for such columns. Represents the set of requested - * columns and tuples as expressed in the physical plan. + * Each column may have structure: a set of referenced names or + * array indicies. Review comment: ```suggestion * array indices. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize
the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377038532
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/physical/resultSet/impl/RowSetTestUtils.java
##
@@ -58,6 +58,10 @@ private RowSetTestUtils() { }
new SchemaPath[] {SchemaPath.STAR_COLUMN});
}
+ public static List projectNone() {
+return new ArrayList<>();
Review comment:
Is it safe to return unmodifiable instance, such as
`Collections.emptyList()` instead?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize the projection parser
KazydubB commented on a change in pull request #1974: DRILL-7574: Generalize
the projection parser
URL: https://github.com/apache/drill/pull/1974#discussion_r377027268
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/resultSet/project/RequestedColumn.java
##
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.physical.resultSet.project;
+
+/**
+ * Plan-time properties of a requested column.Represents
+ * a consolidated view of the set of references to a column.
+ * For example, the project list might contain:
+ * SELECT columns[4], columns[8]
+ * SELECT a.b, a.c
+ * SELECT columns, columns[1]
+ * SELECT a, a.b
+ * In each case, the same column is referenced in different
+ * forms which are consolidated into this abstraction.
+ *
+ * The resulting information is a "pattern": a form of reference
+ * which which a concrete type can be compatible or not. The project
+ * list does not contain sufficient information to definitively pick
+ * a type; it only excludes certain types.
+ *
+ * Depending on the syntax, we can infer if a column must
+ * be an array or map. This is definitive: though we know that
+ * columns of the form above must be an array or a map,
+ * we cannot know if a simple column reference might refer
+ * to an array or map.
+ *
+ * Compatibility Rules
+ *
+ * The pattern given by projection is consistent with certain concrete types
+ * as follows. + means any number of additional qualifiers.
+ *
+ *
+ * TypeConsistent with
+ * Non-repeated MAP
+ * {@code a+} {@code a.b+}
+ * Repeated MAP
+ * {@code a+} {@code a.b+} {@code a[n].b+}>
+ * Non-repeated Scalar
+ * {@code a}
+ * Repeated Scalar
+ * {@code a} {@code a[n]}
+ * Non-repeated DICT
+ * {@code a} {@code a['key']}
+ * Repeated DICT
+ * {@code a} {@code a[n]} {@code a['key']} {@code
a[n]['key']}
+ * Non-repeated LIST
+ * {@code a} {@code a[n]}
+ * Repeated LIST
+ * {@code a} {@code a[n]} {@code a[n][n]}
+ *
+ *
+ * MAP, DICT, UNION and LIST are structured types: projection can reach
+ * into the structure to any number of levels. In such a case, when sufficient
+ * schema information is available, the above rules can be applied recursively
+ * to each level of structure. The recursion can be done in the class for a
+ * DICT (since there is only one child type), but must be external for other
+ * complex types. For MAP, the column can report which specific members
+ * are projected.
+ *
+ * The Text reader allows the {@code columns} column, which allows the
+ * user to specify indexes. This class reports which indexes were actually
+ * selected. Index information is available only at the top level, but
+ * not for 2+ dimensions.
+ */
+public interface RequestedColumn {
+ String name();
+ String fullName();
+ boolean nameEquals(String target);
+
+ boolean isWildcard();
+
+ /**
+ * @return true if this column has no qualifiers. Example:
+ * {@code a}.
+ */
+ boolean isSimple();
+
+ /**
+ * Report whether the projection implies a tuple. Example:
+ * {@code a.b}. Not that this method, and others can only tell
+ * if the projection implies a tuple; the actual column may
+ * be a tuple (MAP), but be projected simply. The map
+ * format also describes a DICT with a VARCHAR key.
+ *
+ * @return true if the column has a map-like projection.
+ */
+ boolean isTuple();
+ RequestedTuple tuple();
+
+ /**
+ * Report whether the first qualifier is an array.
+ * Example: {@code a[1]}. The array format also describes
+ * a DICT with an integer key.
+ * @return true if the column must be an array.
+ */
+ boolean isArray();
+
+ int arrayDims();
+
+ /**
+ * @return true if the column has enumerated indexes.
+ * Example: {@code a[2], a[5]}
+ */
+ boolean hasIndexes();
+
+ /**
+ * Return the maximum index value, if indexes where given.
Review comment:
```suggestion
* Return the maximum index value, if indexes were given.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
