kasakrisz commented on code in PR #6600:
URL: https://github.com/apache/hive/pull/6600#discussion_r3796444295
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java:
##########
@@ -2984,50 +2985,57 @@ private RelNode genTableLogicalPlan(String tableAlias,
QB qb) throws SemanticExc
// NOTE: Table logical schema = Non Partition Cols + Partition Cols +
// Virtual Cols
- // 3.1 Add Column info for non partion cols (Object Inspector fields)
+ // 3.1 Add Column info for cols (Object Inspector fields)
final Deserializer deserializer = tabMetaData.getDeserializer();
StructObjectInspector rowObjectInspector = (StructObjectInspector)
deserializer
.getObjectInspector();
deserializer.handleJobLevelConfiguration(conf);
List<? extends StructField> fields =
rowObjectInspector.getAllStructFieldRefs();
- ColumnInfo colInfo;
- String colName;
- ArrayList<ColumnInfo> cInfoLst = new ArrayList<>();
-
final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
+ int allColCount = tabMetaData.getAllCols().size();
+ List<ColumnInfo> colInfoList = new
ArrayList<>(Collections.nCopies(allColCount, null));
+ Set<String> partColNames = new
HashSet<>(tabMetaData.getPartColNames());
+ ArrayList<ColumnInfo> nonPartitionColumns = new
ArrayList<>(fields.size());
+
for (StructField structField : fields) {
- colName = structField.getFieldName();
- colInfo = new ColumnInfo(
+ String colName = structField.getFieldName();
+ if (partColNames.contains(colName)) {
+ continue;
+ }
+
+ ColumnInfo colInfo = new ColumnInfo(
structField.getFieldName(),
TypeInfoUtils.getTypeInfoFromObjectInspector(structField.getFieldObjectInspector()),
isNullable(colName, nnc, pkc), tableAlias, false);
colInfo.setSkewedCol(isSkewedCol(tableAlias, qb, colName));
- rr.put(tableAlias, colName, colInfo);
- cInfoLst.add(colInfo);
+ colInfoList.set(tabMetaData.getColumnIndexByName(colName), colInfo);
+ nonPartitionColumns.add(colInfo);
}
- // TODO: Fix this
- ArrayList<ColumnInfo> nonPartitionColumns = new
ArrayList<ColumnInfo>(cInfoLst);
- ArrayList<ColumnInfo> partitionColumns = new ArrayList<ColumnInfo>();
// 3.2 Add column info corresponding to partition columns
+ // Normally, the column names in a schema should be unique, but in the
case of Iceberg v1 tables,
+ // updating the partition spec doesn't remove the existing partition
keys, so we can end up with a
+ // partition spec containing multiple columns with the same name.
Review Comment:
```
CREATE TABLE t1_ice(a int, b string, c int) PARTITIONED BY SPEC(a) STORED BY
ICEBERG tblproperties ('format-version'='1');
insert into t1_ice(a, b, c) values
(1, 'aaa', 11),
(2, 'aaa', 12),
(3, 'aab', 13),
(10, 'baa', 14),
(20, 'bac', 15);
alter table t1_ice set partition spec(b);
insert into t1_ice(a, b, c) values
(1, 'aaa', 16);
alter table t1_ice set partition spec(a);
```
While debugging the query
```
explain
select a, b, c from t1_ice group by a, b, c;
```
```
tabMetaData.getStorageHandler().getPartitionKeys(tabMetaData)
```
returns
```
0 = {FieldSchema@32036} "FieldSchema(name:a, type:int, comment:Transform:
void)"
1 = {FieldSchema@32037} "FieldSchema(name:b, type:string, comment:Transform:
void)"
2 = {FieldSchema@32038} "FieldSchema(name:a, type:int, comment:Transform:
identity)"
```
The column `a` is duplicated. The first one has a transform `void` but it is
still in the list.
Then
```
tabMetaData.getPartCols()
```
returns the same.
In case of `'format-version'='2'` tables the returned list contains only
single column `a`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
